数据挖掘概论.课件

1、What is Data Mining ?数据挖掘概论南京航空航天大学信息科学与技术学院皮德常 教授、博导Lots of data is being collected and warehoused Web data, e-commercepurchases at department/grocery storesBank/Credit Card transactionsComputers have become cheaper and more powerfulCompetitive pressure is strong Provide better, customized services

2、 for an edge (e.g. in Customer Relationship Management)Why Mine Data? Commercial ViewpointWhy Mine Data? Scientific ViewpointData collected and stored at enormous speeds (GB/hour)remote sensors on a satellitetelescopes scanning the skiesmicroarrays generating gene expression datascientific simulatio

3、ns generating terabytes of dataTraditional techniques infeasible for raw dataData mining may help scientists in classifying and segmenting data, Mining Large Data Sets - Motivationdata rich but information poor!we are drowning in data, but starving for knowledge!哇！这么多的数据！怎样才能用呢？挖！“Necessity is the m

4、other of invention”Data miningAutomated analysis of massive data setsMining Large Data Sets - MotivationA famous story:跟尿布一起购买最多的商品是啤酒！diapersbeerThe success of GoogleSearch Engine: Analyzing data on the internet to find what meets your demand.Larry Page 1973.3.26 & Sergey Brin 1973.8.21 166亿美元 & 14

5、1亿美元的财产，共享一架波音767 What is Data Mining?Data mining is the non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns from huge volume of data. U. Fayyad, et al. s definition of KDD at KDD96What is (not) Data Mining? What is Data Mining? Certain names a

6、re more prevalent in certain US locations (OBrien, ORurke, OReilly in Boston area) What is not Data Mining? Look up phone number in phone directory Draws ideas from machine learning/AI, pattern recognition, statistics, and database systemsTraditional Techniquesmay be unsuitable due to Enormity of da

7、taHigh dimensionality of dataHeterogeneous, distributed nature of dataOrigins of Data MiningMachine Learning/Pattern RecognitionStatistics/AIData MiningDatabase systemsArchitecture: Typical Data Mining Systemdata cleaning, integration, and selectionDatabase or Data Warehouse ServerData Mining Engine

8、Pattern EvaluationGraphical User InterfaceKnowle-dgeBaseDBDWWWWOther InfoRepositoriesData Mining TasksPredictionUse some variables to predict unknown or future values of other variables.DescriptionFind human-interpretable patterns that describe the data.From Fayyad, et.al. Advances in Knowledge Disc

9、overy and Data Mining, 1996Data Mining Tasks.ClassificationClusteringAssociation Rule DiscoverySequential Pattern DiscoveryRegressionDeviation DetectionClassification ExamplecategoricalcategoricalcontinuousclassTestSetTraining SetModelLearn ClassifierClassification: ApplicationDirect MarketingGoal:

10、Reduce cost of mailing by targeting a set of consumers likely to buy a new cell-phone product.Approach:Use the data for a similar product introduced before. We know which customers decided to buy and which decided otherwise. This buy, dont buy decision forms the class attribute.Collect some related

11、information about the customers.Type of business, where they stay, how much they earn, etc.Use this information as input attributes to learn a classifier model.Clustering DefinitionGiven a set of data points, each having a set of attributes, and a similarity measure among them, find clusters such th

12、atData points in one cluster are more similar to one another.Data points in separate clusters are less similar to one another.ClusteringEuclidean Distance Based Clustering in 3-D space.Intra-cluster distancesare minimizedInter-cluster distancesare maximizedClustering: ApplicationDocument Clustering:

13、Goal: To find groups of documents that are similar to each other based on the important terms appearing in them.Approach: To identify frequently occurring terms in each document. Form a similarity measure based on the frequencies of different terms. Use it to cluster.Gain: Information Retrieval can

14、utilize the clusters to relate a new document or search term to clustered documents.Illustrating Document ClusteringClustering Points: 3204 Articles of Los Angeles Times.Similarity Measure: How many words are common in these documents (after some word filtering).Association Rule DiscoveryGiven a set

15、 of records each of which contain some number of items from a given collection;Produce dependency rules which will predict occurrence of an item based on occurrences of other items.Rules Discovered: Diaper, Milk - BeerAssociation Rule Discovery: Application 1Supermarket shelf management.Goal: To ide

16、ntify items that are bought together by sufficiently many customers.Approach: Process the point-of-sale data collected with barcode scanners to find dependencies among items.A classic rule If a customer buys diaper and milk, then he is very likely to buy beer.So, dont be surprised if you find six-pa

17、cks stacked next to diapers!RegressionPredict a value of a given continuous valued variable based on the values of other variables, assuming a linear or nonlinear model of dependency.Greatly studied in statistics, neural network fields.Examples:Predicting sales amounts of new product based on adveti

18、sing expenditure.Predicting wind velocities as a function of temperature, humidity, air pressure, etc.Time series prediction of stock market indices.Deviation/Anomaly DetectionDetect significant deviations from normal behaviorApplications:Credit Card Fraud DetectionNetwork Intrusion DetectionChallen

19、ges of Data MiningScalabilityDimensionalityComplex and Heterogeneous DataData QualityData Ownership and DistributionPrivacy PreservationStreaming DataMy hope数据挖掘研究已经开展了近15年。推进该技术的广泛应用：1. 企业界已经开始关注数据挖掘技术研究部门应该做什么？2. 自身技术的研究：易用性可用性3. 与应用领域的结合：金融业生物信息学信息检索。飞行器故障诊断与预测、可靠性、My research in recent years1. Mining Acceleration-like Association Rule2. Interior-oriented Intrusion Detection System Based on Multi-agents 3. Fuzzy Clustering Algorithm4. A Fast Trajectory Clustering Algorithm with SamplingMy research in recent years5. An improved C