数据挖掘weka preprocess管理学教授财经信息管理与工程学院

BigDataDataMiningwith—DataPreprocessinginDataPreprocessinginDatacanbeimportedfromafileinvariousformats:ARFF,CSV,C4.5,binaryDatacanalsobereadfromaURLorfromanSQLdatabase(usingJDBC)Pre-processingtoolsinWEKAarecalledWEKAcontainsfiltersDiscretization,normalization,resampling,attributeselection,transformingandcombiningattributes,@relationheart-disease-@attributeagenumeric@attributesex{female,male}@attributechest_pain_type{typ_angina,asympt,non_anginal,atyp_angina}@attributecholesterolnumeric@attributeexercise_induced_angina{no,yes}@attributeclass{present,not_present}@relationheart-disease-@attributeagenumeric@attributesex{female,male}@attributechest_pain_type{typ_angina,asympt,non_anginal,atyp_angina}@attributecholesterolnumeric@attributeexercise_induced_angina{no,yes}@attributeclass{present,not_present}ThisexampleillustratessomeofthebasicdatapreprocessingoperationsthatcanbeperformedusingWEKA.Thesampledatasetusedforthisexample,unlessotherwiseindicated,isthe"bankdata"availableincomma-separatedformat(bank-data.csv).Prof./Dr.Prof./Dr.LIUPageThedatacontainsthefollowingauniqueidentificationageofcustomerinyearsMALE/incomeofcustomeristhecustomermarriednumberofchildrendoesthecustomerownacardoesthecustomerhaveasavingaccountdoesthecustomerhaveacurrentaccountdoesthecustomerhaveamortgagedidthecustomerbuyaPEP(PersonalEquityPlan)afterthelastmailing(YES/NO)InadditiontothenativeARFFdatafileformat,WEKAhasthecapabilitytoreadin".csv"formatfiles.Thisisfortunatesincemanydatabasesorspreadsheetapplicationscansaveorexportdataintoflatfilesinthisformat.Ascanbeseeninthesampledatafile,thefirstrowcontainstheattributenames(separatedbycommas)followedbyeachdatarowwithattributevalueslistedinthesameorder(alsoseparatedbycommas).Infact,onceloadedintoWEKA,thedatasetcanbesavedintoARFFformatInthisweloadthedatasetintoperformaseriesofoperationsusingWEKA'sattributeanddiscretizationfilters,andthenperformassociationruleminingontheresultingdataset.Prof./Dr.LIUProf./Dr.LIUPageInitially(inthePreprocesstab)click"open"andnavigatetothedirectorycontainingthedatafile(.csvor.arff).InthiscasewewillopentheabovedataProf./Dr.Prof./Dr.LIUPageOncethedataisloaded,WEKAwillrecognizetheattributesandduringthescanofthedatawillcomputesomebasicstatisticsoneachattribute.Theleftpanelinthefigureshowsthelistofrecognizedattributes,whilethetoppanelsindicatethenamesofthebaserelation(ortable)andthecurrentworkingrelation(whicharethesameinitially).Prof./Dr.Prof./Dr.LIUClickingonanyattributeintheleftpanelwillshowthebasicstatisticsonthatattribute.Forcategoricalattributes,thefrequencyforeachattributevalueisshown,whileforcontinuousattributeswecanobtainmin,max,mean,standarddeviation,etc.Asanexample,seethefiguresbelowwhichshowtheresultsofselectingthe"age"and"married"attributes,Prof./Dr.Prof./Dr.LIUNotethatthevisualizationintherightbottompanelisaformofcross-tabulationacrosstwoattributes.Forexample,inthefigureabove,thedefaultvisualizationpanelcross-tabulates"married"withthe"pep"attribute(bydefaultthesecondattributeisthelastcolumnofthedataYoucanselectanotherattributeusingthedropdownInoursampledatafile,eachrecordisuniquelyidentifiedbyacustomerid(the"id"attribute).Weneedtoremovethisattributebeforethedatamining选择属性并点击Thiswillremovethe"id"andcreateanewworkingPage(whosenamenowincludesthedetailsofthefilterPageTheresultisdepictedinthefigureProf./Dr.Prof./Dr.LIUItispossiblenowtoapplyadditionalfilterstothenewworkingrelation.Inthisexample,however,wewillsaveourintermediateresultsasseparatedatafilesandtreateachstepasaseparateWEKAsession.TosavethenewworkingrelationasanARFFfile,clickonsavebuttoninthetoppanel.Here,asshowninthe"save"dialogbox,wewillsavethenewrelationinthefileThefigureshowsthetopportionofthenewgeneratedARFFfile(inTextPad).Notethatinthenewdataset,the"id"attributeandallthecorrespondingvaluesintherecordshavebeenremoved.Also,notethatWekahasautomaticallydeterminedthecorrecttypesandvaluesassociatedwiththeattributes,aslistedintheAttributessectionoftheARFFfile.Prof./Dr.LIUProf./Dr.LIUPageProf./Dr.LIUProf./Dr.LIUPageSometechniques,suchasassociationrulemining,canonlybeperformedoncategoricaldata.Thisrequiresperformingdiscretizationonnumericorcontinuousattributes.Thereare3suchattributesinthisdataandInthecaseofthe"children"attributetherangeofpossiblevaluesareonly0,1,2,and3.Prof./Dr.Prof./Dr.LIUPageInthecaseofthe"children"attributetherangeofpossiblevaluesareonly0,1,2,and3.Inthiscase,wehaveoptedforkeepingallofthesevaluesinthedata.Thismeanswesimplydiscretizebyremovingthekeyword"numeric"asthetypeforthe"children"attributeintheARFFandreplacingitwiththesetofdiscreteWedothisdirectlyinourtexteditorasseenintheInthiscase,wehavesavedtheresultingrelationinaseparatefile"bank-data-2.arff".Prof./Dr.LIUProf./Dr.LIUPageWedothisdirectlyinourtexteditorasseenintheInthiscase,wehavesavedtheresultingrelationinaseparatefile"bank-data-2.arff".“children”属性的可能取值为：1,2,and点击”Choose”，选择Prof./Dr.LIUProf./Dr.LIUPageProf./Dr.LIUProf./Dr.LIUPageProf./Dr.LIUProf./Dr.LIUPageWewillrelyonWEKAtoperformdiscretizationonthe"age"and"income"attributes.Inthisexample,wedivideeachoftheseinto3binsTheWEKAdiscretizationfilter,candividetherangesblindly,orusedvariousstatisticaltechniquestoautomaticallydeterminethebestwayofpartitioningthedata.Inthiscase,wewillperformsimpleProf./Dr.LIUProf./Dr.LIUPageFirstwewillloadourfiltereddatasetintoWEKAbyopeningthefile"bank-data-2.arff".Prof./Dr.LIUPageThe"open"dialogboxindepictedProf./Dr.LIUPageIfweselectthe"children"attributeinthisnewdataset,weseethatitisnowacategoricalattributewithfourpossiblediscreteThisisProf./Dr.LIU PageNow,onceagainweactivatetheFilterdialogbox,butthistime,wewillselect"weka.filters.unsupervised.attribute.Discretize"fromthelistProf./Dr.LIUProf./Dr.LIUPageNext,tochangethedefaultsforthisfilter,clickontheboximmediatelytotherightofthe"Choose"button.ThiswillopentheDiscretizeFilterdialogWeentertheindexfortheattributestobe-Inthiscaseweenter1correspondingtoattributeWealsoenter3asthenumberofnotethatitispossibletodiscretizemorethanoneattributeatthesamebyusingalistofattributeindexesProf./Dr.LIUProf./Dr.LIUSincewearedoingsimplebinning,alloftheotheravailableoptionsaresetto"false".ThedialogboxClick"Apply"intheFilterThiswillresultinanewworkingrelationwiththeselectedattributepartitionedinto3bins(seefigure).Prof./Dr.LIU Toexaminetheresults,wesavethenewworkingrelationinthefiledepictedinfigureProf./Dr.LIUProf./Dr.LIUPageLetusnowexaminethenewdatasetusingourtexteditor(inthiscase,TextPad).ThetopportionofthedataisshownintheaboveYoucanobservethatWEKAhasassigneditsownlabelstoeachofthevaluerangesforthediscretizedattribute.Forexample,thelowerrangeinthe"age"attributeislabeled"(-inf-34.333333]"(enclosedinsinglequotesandescapecharacters),whilethemiddlerangeislabeledandsoProf./Dr.Prof./Dr.LIUTheselabelsnowalsoappearinthedatarecordswheretheoriginalagevaluewasinthecorrespondingrange.Prof./Dr.LIUProf./Dr.LIUPageNext,weapplythesameprocesstodiscretizethe"income"attributeinto3bins.Again,Wekaautomaticallyperformsthebinningandreplacesthevaluesinthe"income"columnwiththeappropriateautomaticallygeneratedlabels.Wesavethenewfileinto"bank-data-3.arff",replacingtheolderversion.Clearly,theWEKAlabels,whilereadable,leavemuchtobedesiredasfarasnamingconventionsgo.Prof./Dr.LIUProf