第八章标准规范、工具和发展趋势2

数据挖掘第八章：标准规范、工具和发展趋势本章内容8.1数据挖掘标准与规范8.2数据挖掘工具8.3数据挖掘的研究趋势基本要求：了解数据挖掘在应用中的相关标准规范及未来的研究趋势。8.1数据挖掘标准与规范数据挖掘过程模型是确保数据挖掘工作顺利进行的关键。典型的过程模型有：SPSS的5A模型——评估(Assess)、访问(Access)、分析(Analyze)、行动(Act)、自动化(Automate)SAS的SEMMA模型——采样(Sample)、探索(Explore)、修正(Modify)、建模(Model)、评估(Assess)跨行业数据挖掘过程标准CRISP-DM(CrossIndustryStandardProcessforDataMining)。TwoCrows公司的数据挖掘过程模型，它与正在建立的CRISP-DM有许多相似之处。

数据挖掘相关标准CRISP-DM（交叉行业数据挖掘过程标准，CrossIndustryStandardProcessforDataMining）。SPSS、NCR以及DaimlerChrysler三个在数据挖掘领域经验丰富的公司发起建立一个社团，目的建立数据挖掘方法和过程的标准8.1数据挖掘标准与规范Crisp-DMProjectObjectivesDataUnderstandingDataPreparationModelingEvaluationReportingBackground

Requirements,assumptions,constraints

Terminology

Datamininggoals&successcriteria

ProjectplanInitialDatacollectionreport

Datadescriptionreport

DataExplorationreport

DataqualityreportDatadescriptionreport

Datapre-processingstepsModelingassumption

TestdesignModeldescription

Modelassessment(inc.validation)Assessmentofdataminingresultswith

respecttoobjectivesFinalreport:Summary: Objectives DataMiningprocess DataMiningresults DataMiningassessment

-ConclusionsFuturework(BusinessUnderstanding)(Deployment)WidelyacceptedPROCESSMODELfordataminingProvidesaframeworkfordescribingthemodelingprocessindetail“BESTPRACTICE”BusinessUnderstandingPhaseUnderstandthebusinessobjectivesWhatisthestatusquo?UnderstandbusinessprocessesAssociatedcosts/painDefinethesuccesscriteriaDevelopaglossaryofterms:speakthelanguageCost/BenefitAnalysisCurrentSystemsAssessmentIdentifythekeyactorsMinimum:TheSponsorandtheKeyUserWhatformsshouldtheoutputtake?IntegrationofoutputwithexistingtechnologylandscapeUnderstandmarketnormsandstandards8.1数据挖掘标准与规范BusinessUnderstandingPhaseTaskDecompositionBreakdowntheobjectiveintosub-tasksMapsub-taskstodataminingproblemdefinitionsIdentifyConstraintsResourcesLawe.g.DataProtectionBuildaprojectplanListassumptionsandrisk(technical/financial/business/organisational)factors8.1数据挖掘标准与规范DataUnderstandingPhaseCollectDataWhatarethedatasources?InternalandExternalSources(e.g.Axiom,Experian)Documentreasonsforinclusion/exclusionsDependonadomainexpertAccessibilityissuesArethereissuesregardingdatadistributionacrossdifferentdatabases/legacysystemsWherearethedisconnects?8.1数据挖掘标准与规范DataUnderstandingPhaseDataDescriptionDocumentdataqualityissuesComputebasicstatisticsDataExplorationSimpleunivariatedataplots/distributionsInvestigateattributeinteractionsDataQualityIssuesMissingValues:UnderstanditssourceStrangeDistributions8.1数据挖掘标准与规范DataPreparationPhaseIntegrateDataJoiningmultipledatatablesSummarisation/aggregationofdata

SelectDataAttributesubsetselectionRationaleforInclusion/ExclusionDatasamplingTraining/ValidationandTestsets8.1数据挖掘标准与规范DataPreparationPhaseDataTransformationUsingfunctionssuchaslogFactor/PrincipalComponentsanalysisNormalization/Discretisation/Binarisation

CleanDataHandlingmissingvalues/Outliers

DataConstructionDerivedAttributes

8.1数据挖掘标准与规范TheModelingPhaseBuildModelChooseinitialparametersettingsStudymodelbehaviour:Sensitivityanalysis

AssessthemodelBewareofover-fittingInvestigatetheerrordistribution:IdentifysegmentsofthestatespacewherethemodelislesseffectiveIterativelyadjustparametersettings8.1数据挖掘标准与规范TheEvaluationPhaseValidateModelHumanevaluationofresultsbydomainexpertsEvaluateusefulnessofresultsfrombusinessperspectiveDefinecontrolgroupsCalculateliftcurvesExpectedReturnonInvestmentReviewProcessDeterminenextstepsPotentialfordeploymentDeploymentarchitectureMetricsforsuccessofdeployment8.1数据挖掘标准与规范PMML（预测模型标记语言，PredictiveModelMarkupLanguage）。数据挖掘应用往往需要多种类型的数据挖掘软件、算法协同运行，这就要求对挖掘出的模型能够很好地继承、复用与集成。DMG（TheDataMiningGroup，DMG）提出PMML语言。PMML最新版本为4.1，支持16种数据挖掘模型，包括：

AssociationModel（关联规则）、BaselineModel（基准模型）、ClusteringModel（聚类模型）、GeneralRegressionModel（回归模型）、MiningModel（组合模型）、NaiveBayesModel（朴素贝叶斯）、

NearestNeighborModel（最近邻模型）NeuralNetwork（神经网络）、RegressionModel（线性、多项式、对数三种回归模型）、RuleSetModel（规则集）、SequenceModel（序列模式）、Scorecard、TimeSeriesModel、SupportVectorMachineModel（支持向量机）、TextModel（文本模型）、TreeModel（决策树）8.1数据挖掘标准与规范PMML的模型定义由以下几部分组成：8.1数据挖掘标准与规范TheheaderelementcontainsgeneralinformationaboutthePMMLdocument,suchascopyrightformationforthemodel,itsdescription,andinformationabouttheapplicationusedtogeneratethemodelsuchasnameandversion.8.1数据挖掘标准与规范<PMMLversion="3.2"...<Headercopyright="Copyright(c)2009Togaware"description="RPartDecisionTree"><Extensionname="timestamp"value="2009-02-1506:51:50"extender="Rattle"/><Extensionname="description"value="iristree"extender="Rattle"/><Applicationname="Rattle/PMML"version="1.2.7"/></Header>Thedatadictionaryrecordsinformationaboutthedatafieldsfromwhichthemodelwasbuilt.8.1数据挖掘标准与规范<DataDictionarynumberOfFields="5"><DataFieldname="Species"...<Valuevalue="setosa"/><Valuevalue="versicolor"/><Valuevalue="virginica"/><DataFieldname="Sepal.Length"optype="continuous"dataType="double"/></DataField>DataTransformations:transformationsallowforthemappingofuserdataintoamoredesirableformtobeusedbytheminingmodel.PMMLdefinesseveralkindsofsimpledatatransformations.Normalization:mapvaluestonumbers,theinputcanbecontinuousordiscrete.Discretization:mapcontinuousvaluestodiscretevalues.Valuemapping:mapdiscretevaluestodiscretevalues.Functions(customandbuilt-in):deriveavaluebyapplyingafunctiontooneormoreparameters.Aggregation:usedtosummarizeorcollectgroupsofvalues.8.1数据挖掘标准与规范Model:containsthedefinitionofthedataminingmodel.ModelName(attributemodelName)AlgorithmName(attributealgorithmName)NumberofLayers(attributenumberOfLayers)MiningSchema:listsallfieldsusedinthemodel.Name:mustrefertoafieldinthedatadictionaryUsagetype:definesthewayafieldistobeusedinthemodel.Typicalvaluesare:active,predicted,andsupplementary.Predictedfieldsarethosewhosevaluesarepredictedbythemodel.OutlierTreatment:definestheoutliertreatmenttobeuse.MissingValueReplacementPolicy:ifthisattributeisspecifiedthenamissingvalueisautomaticallyreplacedbythegivenvalues.MissingValueTreatment:indicateshowthemissingvaluereplacementwasderived.8.1数据挖掘标准与规范Targets:allowforpost-processingofthepredictedvalueintheformatofscalingiftheoutputofthemodeliscontinuous.

8.1数据挖掘标准与规范PMMLExample:AssociationRule：8.1数据挖掘标准与规范t1:Cracker,Coke,Watert2:Cracker,Watert3:Cracker,Watert4:Cracker,Coke,WaterModelattributesItemsPMMLExample:AssociationRule：8.1数据挖掘标准与规范t1:Cracker,Coke,Watert2:Cracker,Watert3:Cracker,Watert4:Cracker,Coke,Water<AssocItemsetid="1"support="1.0"numberOfItems="1"/><AssocItemRef

itemRef="1"/></AssocItemset><AssocItemsetid="2"support="1.0"numberOfItems="1"/><AssocItemRef

itemRef="3"/></AssocItemset><!--andonefrequentitemsetwithtwoitems.--><AssocItemsetid="3"support="1.0"numberOfItems="2"/><AssocItemRef

itemRef="1"/><AssocItemRef

itemRef="3"/></AssocItemset><!--Tworulessatisfytherequirements--><AssocRulesupport="1.0"confidence="1.0"antecedent="1"consequent="2"/><AssocRulesupport="1.0"confidence="1.0"antecedent="2"consequent="1"/></AssociationModel></PMML>ItemSetsAssociationRulesJDM（JavaDataMiningAPI）。旨在提供一个访问数据挖掘工具的标准API，支持数据挖掘模型的建立、使用，数据及元数据的创建、存储、访问及维护，从而使得Java应用程序能够能够方便集成数据挖掘技术。8.1数据挖掘标准与规范SemanticWeb相关标准TimBerners-Lee在XML2000会议报告中首次提出了语义Web的层次模型（LayerCake）。其特点在与：基于XML和RDF/RDFS，构建本体和逻辑推理规则，以完成基于语义的知识表示和推理，从而为计算机所理解和处理。8.1数据挖掘标准与规范第一层是Unicode（统一编码）和URI（UniformResourceIdentifier，统一资源标识器）。UNICODE于1993年成为国际标准组织ISO的一项国际标准ISO/IEC10646，其宗旨是全球所有文种统一编码。URI包含三个部分：被用来访问资源的统一命名规则分配体系、资源宿主机器的名称、路径形式的资源名称。与URL本不同的是，URI只是一个标识符，不直接提供访问资源的方法。8.1数据挖掘标准与规范第二层是XML（EXtensibleMarkupLanguage）。XML具有简单、自描述、可扩展的特点，并且实现了内容、结构和表现三者的分离，因而，更适合于数据表示和交换。XMLSchema中的约束主要用于XML文档的结构合法性验证。第三层是RDF（ResourceDescriptionFramework，资源描述框架）。元数据层。RDF是建立在XML上的元数据描述与交换框架，以“资源（Resource）－属性（Property）－属性值（PropertyValue）”的形式描述对象。一个例子8.1数据挖掘标准与规范8.1数据挖掘标准与规范8.1数据挖掘标准与规范第四层是RDF-S（RDFSchema）。RDF-S是对RDF的扩展，是RDF的词汇描述语言（VocabularyDescriptionLanguage），用于定义RDF资源描述文件中出现的词汇。第五层是本体（Ontology）和规则（Rule）。领域知识层。OWL用于明确表示词汇体系中的术语及术语间的关系，在词义和语义的表达来说，OWL有更强的表达能力。规则用于描述领域知识中的前提和结论。SPARQL（SimpleProtocolandRDFQueryLanguage）是W3C推荐的用于对RDF数据查询的语言和协议。8.1数据挖掘标准与规范本章内容8.1数据挖掘标准与规范8.2数据挖掘工具8.3数据挖掘的研究趋势Freeopen-sourcedataminingsoftwareandapplicationsGATE:a

naturallanguageprocessing

andlanguageengineeringtool.Orange:Acomponent-baseddataminingand

machinelearning

softwaresuitewritteninthe

Python

language.R:A

programminglanguage

andsoftwareenvironmentforstatisticalcomputing,datamining,andgraphics.RapidMiner:Anenvironmentfor

machinelearning

anddataminingexperiments.UIMA:TheUIMA(UnstructuredInformationManagementArchitecture)isacomponentframeworkforanalyzingunstructuredcontentsuchastext,audioandvideo–originallydevelopedbyIBM.Weka:Asuiteofmachinelearningsoftwareapplicationswritteninthe

Java

programminglanguage.8.2数据挖掘工具Commercialdata-miningsoftwareandapplicationsIBMSPSSModeler:dataminingsoftwareprovidedbyIBM.MicrosoftAnalysisServices:dataminingsoftwareprovidedbyMicrosoft.OracleDataMining:dataminingsoftwarebyOracle.SASEnterpriseMiner:dataminingsoftwareprovidedbytheSASInstitute.STATISTICADataMiner:dataminingsoftwareprovidedbyStatSoft.8.2数据挖掘工具8.2数据挖掘工具MainFeatures49datapreprocessingtools76classification/regressionalgorithms8clusteringalgorithms3algorithmsforfindingassociationrules15attribute/subsetevaluators+10searchalgorithmsforfeatureselectionMainGUI“TheExplorer”(exploratorydataanalysis)“TheExperimenter”(experimentalenvironment)“TheKnowledgeFlow”(newprocessmodelinspiredinterface)8.2数据挖掘工具WEKAonlydealswith“flat”files8.2数据挖掘工具@relationheart-disease-simplified@attributeagenumeric@attribu