{"id":455,"date":"2012-12-15T20:02:00","date_gmt":"2012-12-15T12:02:00","guid":{"rendered":"http:\/\/note.systw.net\/note\/?p=455"},"modified":"2023-11-02T20:03:29","modified_gmt":"2023-11-02T12:03:29","slug":"data-preprocessing","status":"publish","type":"post","link":"https:\/\/systw.net\/note\/archives\/455","title":{"rendered":"Data Preprocessing"},"content":{"rendered":"\n

data preprocessing<\/strong>
\u5e38\u898b\u4efb\u52d9\u6709:
data cleaning:\u89e3\u6c7adirty data
data integration:\u5c07\u591a\u500b\u4f86\u6e90\u8cc7\u6599\u6574\u5408
data transformation:\u6b63\u898f\u5316,\u5c07\u6240\u6709\u8cc7\u6599\u8abf\u5230\u56fa\u5b9a\u7bc4\u570d
data reduction:\u5c07\u76f8\u540c\u7684\u8cc7\u6599\u522a\u6389,\u6216\u5c07\u8cc7\u6599\u53d6\u6a23
data discretization and concept hierarchy generation<\/p>\n\n\n\n

……………………………………………………………………………………………………..<\/p>\n\n\n\n

data cleaning<\/strong>
data cleaning\u662f\u8cc7\u6599\u5009\u5132\u4e09\u5927\u554f\u984c\u7684\u5176\u4e2d\u4e00\u500b,\u4e5f\u662f\u6700\u91cd\u8981\u7684\u554f\u984c<\/p>\n\n\n\n

data in the real world is dirty<\/strong>
\u56e0\u70badata\u6703\u5305\u62ec
\u3000incomplete data
\u3000noisy data
\u3000inconsistent data
\u3000duplicate records
ps:
no quality data,no quality mining results<\/p>\n\n\n\n

\u70ba\u89e3\u6c7a\u4e0a\u8ff0dirty data,\u4e3b\u8981\u4efb\u52d9\u5982\u4e0b
\u8655\u7406missing data<\/strong>\uff1a\u6709\u4e9b\u8cc7\u6599\u6709\u907a\u6f0f,\u8cc7\u6599\u4e0d\u5b8c\u6574
\u8655\u7406noisy data<\/strong>\uff1a\u8cc7\u6599\u6709\u554f\u984c,\u53ef\u80fd\u5728\u8f49\u6a94\u6642\u6216\u8f38\u5165\u6642\u932f\u8aa4
\u5075\u6e2cdata discrepancy<\/strong><\/p>\n\n\n\n

\u5075\u6e2cdata discrepancy\u65b9\u6cd5\u4e3b\u8981\u6709
\u3000\u4f7f\u7528metadata:\u900f\u904edomain,range,dependency,distribution\u7684\u8a08\u7b97
\u3000\u6aa2\u67e5field overloading
\u3000\u6aa2\u67e5uniqueness rule,consecutive rule , null rule
\u3000\u4f7f\u7528\u5546\u696d\u5de5\u5177\u505adata scrubbing\u53cadata auditing<\/p>\n\n\n\n

\u8655\u7406missing data\u4e3b\u8981\u65b9\u5f0f\u5982\u4e0b
ignore the tuple<\/strong>:\u5ffd\u7565\u6389\u9019\u985e\u8cc7\u6599
fill by manually<\/strong>:\u7528\u624b\u52d5\u586b\u503c\u65b9\u5f0f\u88dc\u503c
fill by automatically<\/strong>:\u6839\u64da\u6307\u5b9a\u898f\u5247\u81ea\u52d5\u586b\u503c,\u5e38\u898b\u898f\u5247\u5982\u4e0b
\u3000\u586b\u5165unknown,\u4ee5\u8996\u70ba\u53e6\u4e00\u500b\u503c
\u3000\u586b\u5165attribute mean\u7684\u503c
\u3000\u3000by all:missing data\u4ee5\u8a72attribute mean\u4ee3\u66ff
\u3000\u3000by class:\u5c07\u8cc7\u6599\u5206\u70ba\u4e0d\u540cclass,\u8a72class\u7684missing data\u4ee5\u8a72class \u7684attribute mean\u4ee3\u66ff
\u3000\u586b\u5165bayesian formula\u6216decision tree\u7522\u751f\u7684\u503c<\/p>\n\n\n\n

\u8655\u7406noisy data\u4e3b\u8981\u65b9\u5f0f\u5982\u4e0b
binning<\/strong>
regression(\u8ff4\u6b78\u5206\u6790)<\/strong>
clustering<\/strong>:\u627e\u51fa\u5c07outlier value\u4e26\u6392\u9664<\/p>\n\n\n\n

ps:
binning(\u5207\u5272\u6cd5)<\/strong>
1 sort data and partition into bins
\u5207\u5272\u6210M\u4efd\u6642,\u6709\u4ee5\u4e0b\u5169\u7a2e\u8655\u7406\u65b9\u5f0f
equal-depth(frequency):\u6bcfM\u4efd\u7684\u8cc7\u6599\u91cf\u8981\u76f8\u540c(\u6578\u503c\u5bec\u5ea6\u53ef\u4e0d\u540c)
euqal-width(distance):\u6bcfM\u4efd\u7684\u6578\u503c\u5bec\u5ea6\u8981\u76f8\u540c(\u8cc7\u6599\u91cf\u53ef\u4e0d\u540c)
2 smooth
\u5c07\u4e2d\u9593\u7684\u503c\u900f\u904emean,median,boundaries,…\u7b49\u65b9\u6cd5\u91cd\u65b0\u8a08\u7b97
ex:
data set:4,8,9,15,21,21,24,25,26,28,29,34
1 partition into bins
\u82e5\u8981\u7528equal-width\u52073\u4efd
\u5247\u6bcf\u4efd\u7684\u6578\u503c\u5bec\u5ea6\u9700\u70ba(34-4+1)\/12=10.3
\u3000bin1:4,8,9
\u3000bin2:15,21,21,24
\u3000bin3:25,26,28,29,34
2 smooth
smooth by mean
\u3000bin1:8,8,8
\u3000bin2:21,21,21,21
\u3000bin3:28,28,28,28,28
smooth by boundaries
\u3000bin1:4,9,9
\u3000bin2:15,24,24,24
\u3000bin3:25,25,25,25,34<\/p>\n\n\n\n


……………………………………………<\/p>\n\n\n\n


data integration<\/strong>
combines data from multiple sources into a coherent store<\/p>\n\n\n\n

\u6574\u5408\u53ef\u80fd\u6703\u78b0\u5230\u554f\u984c
entity identification<\/strong>:\u4e0d\u540c\u540d\u7a31\u7684\u8cc7\u6599\u4f46\u4ee3\u8868\u7684\u610f\u601d\u662f\u76f8\u540c\u7684,ex:tw=taiwan
\u4e0d\u540c\u55ae\u4f4d<\/strong>:\u5728integration\u8981\u8f49\u6210\u76f8\u540c\u55ae\u4f4d , ex:byte vs bit
redundancy data<\/strong>:\u5e38\u898b\u6709\u4ee5\u4e0b
\u3000object identification:\u76f8\u540cattribute\u4f46\u5728\u4e0d\u540c\u4f86\u6e90\u6709\u4e0d\u540c\u540d\u7a31 ex:\u6b04\u4f4dsrcaddr = \u6b04\u4f4dsource \u3000
\u3000derivable data:\u8a72attribute\u7684\u8cc7\u6599\u53ef\u88ab\u53e6\u4e00\u500battribute\u6240\u63a8\u5c0e ex: \u6b04\u4f4dbpp = \u6b04\u4f4dbyte\/\u6b04\u4f4dpacket
ps:
redundancy data\u53ef\u900f\u904ecorrelation analysis\u5075\u6e2c\u51fa\u4f86
\u82e5\u662fnumerical data\u53ef\u4ee5\u7528correlation coefficient\u5075\u6e2c\u51fa\u4f86
\u82e5\u662fcategorical data\u53ef\u4ee5\u7528chi-square test\u5075\u6e2c\u51fa\u4f86<\/p>\n\n\n\n

ps:
chi-square test<\/strong>
x2=segmal( (observed-expected)^2\/expected )
x2\u7684\u503c\u8d8a\u5927\u76f8\u95dc\u6027\u8d8a\u9ad8
ex:
\u6709\u4e00\u5047\u8a2d\u5ba3\u7a31p2p\u7684\u4f7f\u7528\u548c\u662f\u5426\u4e2d\u6bd2\u6c92\u95dc\u4fc2(H0)
\u70ba\u4e86\u6aa2\u5b9a\u4e00\u5047\u8a2d,\u5be6\u969b\u62bd\u6a23\u8abf\u67e5\u7d50\u679c,observed\u5982\u4e0b<\/p>\n\n\n\n

 <\/td>\u4f7f\u7528p2p <\/td>\u4e0d\u4f7f\u7528p2p <\/td>sum(row) <\/td><\/tr>
\u4e2d\u6bd2 <\/td>250(90) <\/td>200(360) <\/td>450 <\/td><\/tr>
\u6c92\u4e2d\u6bd2 <\/td>50(210) <\/td>1000(840) <\/td>1050 <\/td><\/tr>
sum(column) <\/td> 300 <\/td> 1200 <\/td> 1500 <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

ps:
expected(90)=450*300\/1500
x2=(250-90)^2\/90+(200-360)^2\/360+(50-210)^2\/210+(1000-840)^2\/840=507.93
\u82e5\u8981\u62d2\u7d55H0(\u63a8\u7ffb\u539f\u672c\u5047\u8a2d\u7684\u5ba3\u7a31)\u5247\u9700\u5927\u65bc\u986f\u8457\u6c34\u6e96,
\u82e5\u986f\u8457\u6c34\u6e96\u70ba0.001,\u5728\u81ea\u7531\u5ea61\u7684\u60c5\u6cc1\u4e0b\u70ba10.828<\/strong>
ps:
\u81ea\u7531\u5ea6=(row-1)(column-1)=(2-1)(2-1)=1
x2>10.828,\u62d2\u7d55H0\u63a8\u7ffb\u539f\u5047\u8a2d\u6240\u5ba3\u7a31,\u56e0\u6b64\u63db\u53e5\u8a71\u8aaa\u672c\u6b21\u62bd\u6a23\u8aaa\u660ep2p\u4f7f\u7528\u548c\u4e2d\u6bd2\u662f\u6709\u95dc\u4fc2\u7684<\/p>\n\n\n\n

……………………………………<\/p>\n\n\n\n


data transformation
<\/strong>\u6b63\u898f\u5316,\u5c07\u6240\u6709\u8cc7\u6599\u8abf\u5230\u56fa\u5b9a\u7bc4\u570d<\/p>\n\n\n\n

\u5e38\u898b\u7684\u4efb\u52d9\u6709
aggregation\/summary<\/strong>:\u4e00\u822c\u5efa\u7acbdata cube\u4e5f\u6703\u5305\u542b\u9019\u500b\u90e8\u4efd
\u3000ex:\u6bcf\u5929\u7684\u92b7\u552e\u88ab\u805a\u96c6\u6210\u4e00\u500b\u6708\u6216\u4e00\u5e74\u7684\u92b7\u552e\u8cc7\u6599
\u3000ex:\u5c07\u6bcf\u5929\u7684\u6d41\u91cf\u8cc7\u6599aggregation\u6210\u6bcf\u6708\u6216\u6bcf\u9031\u7684\u8cc7\u6599
\u3000ex:\u5c07flow\u7684\u6578\u64daaggregation\u6210flowg\u7684\u6578\u64da,\u5728aggregation\u6210ip\u7684\u6578\u64da
generalization<\/strong>:concept hierarchy climbing ,
\u3000ex:categorical\u5982\u4f4f\u5740\u53ef\u4ee5\u88ab\u62c6\u6210\u57ce\u5e02,\u9053\u8def,\u9580\u724c\u865f\u78bc
\u3000ex:numerical\u5982\u5e74\u9f61\u53ef\u4ee5\u88ab\u62c6\u6210\u5e74\u8f15\u4eba,\u4e2d\u5e74\u4eba,\u9577\u8005
\u3000ex:ip,ip\u6240\u5728\u7684lan,\u8a72lan\u6240\u5728\u7684\u570b\u5bb6
normalization<\/strong>:\u5c07\u6578\u503c\u8f49\u63db\u6210\u4e00\u500brange,\u5e38\u898b\u65b9\u6cd5\u5982\u4e0b
\u3000min-max normalization:\u516c\u5f0f\u70ba((v-min)\/( max-min)) * (new_max-new_min)+new_min
\u3000z-score normalization:\u900f\u904e\u7d71\u8a08\u65b9\u5f0f\u5c07\u6578\u64da\u8f49\u6210z-score,\u516c\u5f0f\u70ba (v-mean)\/s
\u3000normalization by decimal scaling(\u4e0d\u5efa\u8b70\u7528)
ps:min-max normalization\u7121\u6cd5\u627e\u5230outlier\uff0c\u56e0\u70ba\u88ab\u5206\u4f48\u57280-1\u4e4b\u9593
ex:
byte min=12000,byte max=98000,
xi byte=73600,\u4f7f\u7528min-max normalization\u8f49\u63db\u62100-1\u4e4b\u9593\u7684\u6578\u503c
\u5247\u65b0\u503c=((73600-12000)\/( 98000-12000)) * (1-0)+0=0.71627907<\/p>\n\n\n\n


……………………………………………<\/p>\n\n\n\n


data reduction
<\/strong>\u5c07\u76f8\u540c\u7684\u8cc7\u6599\u522a\u6389,\u6216\u5c07\u8cc7\u6599\u53d6\u6a23<\/p>\n\n\n\n

\u5e38\u898b\u7b56\u7565\u6709
data cube aggregation
dimensionality reduction :
data compression:\u901a\u5e38\u7528\u65bc\u591a\u5a92\u9ad4
numerosity reduction
discretization and concept hierarchy generation:\u5c07raw data\u8f49\u63db\u6210higher conceptual level<\/p>\n\n\n\n


dimensionality reduction<\/strong>
\u76ee\u5730:\u79fb\u9664\u4e0d\u91cd\u8981\u7684dimensionality,\u627e\u51fa\u6240\u6709\u96c6\u5408\u4e2d\u9054\u5230\u76ee\u6a19\u7684\u6700\u5c0f\u96c6\u5408
\u5e38\u7528\u65b9\u6cd5\u5305\u62ec
decision-tree induction
heuristic feature selection method
principal component analysis:\u5c07\u6578\u503c\u578b\u8cc7\u6599\u8b8a\u6210\u5411\u91cf,\u7528\u5728\u7dad\u5ea6\u592a\u5927\u6642,\u4e3b\u8981\u662f\u5f9en\u500b\u7dad\u5ea6\u4e2d\u627e\u51fak\u500b\u8cc7\u6599\u6700\u4f73\u4ee3\u8868\u7684\u7dad\u5ea6<\/p>\n\n\n\n


numerosity reduction<\/strong>
\u76ee\u5730:\u5c07\u6578\u64da\u91cf\u8b8a\u5c11
parametric method:
\u3000\u5305\u62eclinear regression,multiple regression,log-linear model
non-parametric method:\u5305\u62ec
\u3000histograms:\u5c07\u8cc7\u6599\u5207\u5272,\u6bcf\u4efd\u5728\u7b97\u51fa\u5e73\u5747\u6578\u4f86\u4ee3\u8868\u8cc7\u6599
\u3000\u3000\u5e38\u898b\u5207\u5272\u65b9\u6cd5\u5305\u62ec:equal-width,equal-frequency,v-optimal(\u7528\u8b8a\u7570\u6578\u5207\u5272),maxdiff
\u3000clustering:\u8cc7\u6599\u4ee5\u7fa4\u8868\u793a,\u5c07outlier\u7684\u8cc7\u6599\u6392\u9664
\u3000sampling:\u9078\u64c7\u6709\u4ee3\u8868\u6027\u7684\u8cc7\u6599
\u3000\u3000SRSWOR:\u62bd\u6a23\u8d8a\u591a,\u6709\u9650\u6bcd\u9ad4\u8cc7\u6599\u6703\u8d8a\u4f86\u8d8a\u5c11
\u3000\u3000SRSWR:\u62bd\u6a23\u904e\u7a0b\u4e2d,\u6709\u9650\u6bcd\u9ad4\u4fdd\u6301\u4e0d\u8b8a
\u3000\u3000stratified sampling:\u5148\u5206\u7fa4,\u5728\u4f9d\u5404\u7fa4\u6bd4\u4f8b\u53d6\u6a23
\u3000\u3000cluster sampling:\u5148\u5206\u7fa4,\u5728\u53d6\u5176\u4e2d\u5e7e\u7fa4\u505a\u70ba\u6a23\u672c<\/p>\n\n\n\n

……………………….<\/p>\n\n\n\n

data discretization and concept hierarchy generation<\/strong>
concept hierarchy<\/strong>
\u3000\u5c07low level\u8cc7\u6599\u4ee5high level\u8cc7\u6599\u53d6\u4ee3 ex:\u5e74\u9f61\u6578\u64da\u4ee5{\u8001,\u5c11,\u5e7c}3\u500bvalue\u8868\u793a
discretization<\/strong>
\u3000\u5c07continuous\u8cc7\u6599\u5206\u6210n\u500b\u5340\u9593
\u3000\u7528\u9014:\u53ef\u7528\u4f86data reduction\uff0c\u6216\u505a\u66f4\u9032\u4e00\u6b65\u7684\u5206\u6790
\u3000\u4f9d\u662f\u5426\u6709\u8a13\u7df4\u8cc7\u6599\u53ef\u5206\u70ba supervised \u548c unsupervised
\u3000\u4e5f\u53ef\u5206\u70ba split(top-down) \u548c merge(bottom-up)
ps:
3\u7a2e\u5c6c\u578b\u985e\u578b
nominal(\u540d\u76ee):values from an unordered set ,ex:color,protocol
ordinal(\u6709\u6392\u540d\u7684):values from an ordered ,ex:virus rank
continuous(\u5be6\u6578\u7684):real number ,ex: byte<\/p>\n\n\n\n

\u7528\u65bcnumerical data
top-down split,<\/strong>
\u3000binning:\u5c6c\u65bcunsupervised
\u3000histogram analysis: \u5c6c\u65bcunsupervised
\u3000natural partitioning\/3-4-5 rule:\u5c6c\u65bcunsupervised
\u3000entropy-based discretization(\u8907\u96dc\u5ea6\u5206\u6790):\u5148\u5047\u8a2d\u4e00\u500b\u5207\u5272\u9ede,\u5728\u8a08\u7b97\u8a72\u9ede\u5de6\u53f3\u5169\u908a\u7684\u503c\u4ee5\u8a55\u4f30\u8a72\u9ede\u662f\u5426\u5c6c\u65bc\u597d\u7684\u5206\u5272\u9ede,\u53cd\u8986\u4f5c\u696d\u76f4\u5230\u627e\u5230\u6700\u597d\u7684\u5206\u5272\u9ede,\u5c6c\u65bcsupervised
bottom-up merge<\/strong>
\u3000chi-squre test analysis:\u5c6c\u65bcunsupervised
top-down split or bottom-up merge<\/strong>
\u3000clustering analysis:\u5c6c\u65bcunsupervised<\/p>\n\n\n\n

\u7528\u65bccategorical data
by schema level<\/strong>
ex:\u8868\u9054address\u6b04\u4f4d\u6709country,city,street,\u5247\u6839\u64da\u5224\u65b7\u67b6\u69cb\u61c9\u70ba(low-level)street < city < country (high-level)
by explicit data grouping<\/strong>
ex:{ 10.1.0.0\/16 , 10.2.0.0\/16 }\u5c6c\u65bcsales network,{ 172.16.1.0\/24 , 172.16.2.0\/24 }\u5c6c\u65bcresearch network,
\u8a08\u7b97\u6bcf\u500b\u7dad\u5ea6\u7684distinct value(\u4e0d\u91cd\u8907\u7684\u503c)<\/strong>
distinct value\u8d8a\u591a\u7684\u7dad\u5ea6\u901a\u5e38\u6703\u653e\u5728low-level,distinct value\u8d8a\u5c11\u7684\u7dad\u5ea6\u901a\u5e38\u6703\u653e\u5728high-level
ex: \u7db2\u8def\u4f4d\u7f6e\u7531 city,ip\u8868\u9054,distinct value\u5206\u5225\u5f97\u5230\u70ba30\u548c1000,\u5247city\u5c6c\u65bchigh-level,ip\u5c6c\u65bclow-level<\/p>\n","protected":false},"excerpt":{"rendered":"

data preprocessing\u5e38\u898b\u4efb\u52d9\u6709:data c …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"","fifu_image_alt":"","_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[13],"tags":[],"class_list":["post-455","post","type-post","status-publish","format-standard","hentry","category-dataanalysis"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/systw.net\/note\/wp-json\/wp\/v2\/posts\/455","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/systw.net\/note\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/systw.net\/note\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/systw.net\/note\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/systw.net\/note\/wp-json\/wp\/v2\/comments?post=455"}],"version-history":[{"count":0,"href":"https:\/\/systw.net\/note\/wp-json\/wp\/v2\/posts\/455\/revisions"}],"wp:attachment":[{"href":"https:\/\/systw.net\/note\/wp-json\/wp\/v2\/media?parent=455"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/systw.net\/note\/wp-json\/wp\/v2\/categories?post=455"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/systw.net\/note\/wp-json\/wp\/v2\/tags?post=455"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}