浅探关节镜下盘状半月板损伤的治疗课件

CS276B

TextInformationRetrieval,Mining,andExploitationLecture523January2003CS276B

TextInformationRetrie1RecapRecap2Today’stopicsFeatureselectionfortextclassificationMeasuringclassificationperformanceNearestneighborcategorizationToday’stopicsFeatureselectio3FeatureSelection:Why?Textcollectionshavealargenumberoffeatures10,000–1,000,000uniquewords–andmoreMakeusingaparticularclassifierfeasibleSomeclassifierscan’tdealwith100,000soffeat’sReducetrainingtimeTrainingtimeforsomemethodsisquadraticorworseinthenumberoffeatures(e.g.,logisticregression)ImprovegeneralizationEliminatenoisefeaturesAvoidoverfittingFeatureSelection:Why?Textco4Recap:FeatureReductionStandardwaysofreducingfeaturespacefortextStemmingLaugh,laughs,laughing,laughed->laughStopwordremovalE.g.,eliminateallprepositionsConversiontolowercaseTokenizationBreakonallspecialcharacters:fire-fighter->fire,fighterRecap:FeatureReductionStanda5FeatureSelectionYangandPedersen1997ComparisonofdifferentselectioncriteriaDF–documentfrequencyIG–informationgainMI–mutualinformationCHI–chisquareCommonstrategyComputestatisticforeachtermKeepntermswithhighestvalueofthisstatisticFeatureSelectionYangandPede6InformationGainInformationGain7(Pointwise)MutualInformation(Pointwise)MutualInformation8Chi-SquareTermpresentTermabsentDocumentbelongstocategoryABDocumentdoesnotbelongtocategoryCDX^2=N(AD-BC)^2/((A+B)(A+C)(B+D)(C+D))UseeithermaximumoraverageX^2Valueforcompleteindependence?Chi-SquareTermpresentTermabs9DocumentFrequencyNumberofdocumentsatermoccursinIssometimesusedforeliminatingbothveryfrequentandveryinfrequenttermsHowisdocumentfrequencymeasuredifferentfromtheother3measures?DocumentFrequencyNumberofdo10Yang&Pedersen:ExperimentsTwoclassificationmethodskNN(knearestneighbors;morelater)LinearLeastSquaresFitRegressionmethodCollectionsReuters-2217392categories16,000uniquetermsOhsumed:subsetofmedline14,000categories72,000uniquetermsLtctermweightingYang&Pedersen:ExperimentsTwo11Yang&Pedersen:ExperimentsChoosefeaturesetsizePreprocesscollection,discardingnon-selectedfeatures/wordsApplytermweighting->featurevectorforeachdocumentTrainclassifierontrainingsetEvaluateclassifierontestsetYang&Pedersen:ExperimentsChoo12浅探关节镜下盘状半月板损伤的治疗课件13DiscussionYoucaneliminate90%offeaturesforIG,DF,andCHIwithoutdecreasingperformance.Infact,performanceincreaseswithfewerfeaturesforIG,DF,andCHI.Mutualinformationisverysensitivetosmallcounts.IGdoesbestwithsmallestnumberoffeatures.Documentfrequencyisclosetooptimal.Byfarthesimplestfeatureselectionmethod.SimilarresultsforLLSF(regression).DiscussionYoucaneliminate9014ResultsWhyisselectingcommontermsagoodstrategy?ResultsWhyisselectingcommon15IG,DF,CHIAreCorrelated.IG,DF,CHIAreCorrelated.16InformationGainvsMutualInformationInformationgainissimilartoMIforrandomvariablesIndependence?Incontrast,pointwiseMIignoresnon-occurrenceoftermsE.g.,forcompletedependence,youget:P(AB)/(P(A)P(B))=1/P(A)–largerforraretermsthanforfrequenttermsYang&Pedersen:PointwiseMIfavorsraretermsInformationGainvsMutualInf17FeatureSelection:

OtherConsiderationsGenericvsClass-SpecificCompletelygeneric(class-independent)SeparatefeaturesetforeachclassMixed(alaYang&Pedersen)MaintainabilityovertimeIsaggressivefeaturesselectiongoodorbadforrobustnessovertime?Ideal:OptimalfeaturesselectedaspartoftrainingFeatureSelection:

OtherConsi18Yang&Pedersen:LimitationsDon’tlookatclassspecificfeatureselectionDon’tlookatmethodsthatcan’thandlehigh-dimensionalspacesEvaluatecategoryranking(asopposedtoclassificationaccuracy)Yang&Pedersen:LimitationsDon’19FeatureSelection:OtherMethodsStepwisetermselectionForwardBackwardExpensive:needtodon^2iterationsoftrainingTermclusteringDimensionreduction:PCA/SVDFeatureSelection:OtherMetho20WordRep.vs.DimensionReductionWordrepresentations:onedimensionforeachword(binary,count,orweight)Dimensionreduction:eachdimensionisauniquelinearcombinationofallwords(linearcase)Dimensionreductionisgoodforgenerictopics(“politics”),badforspecificclasses(“ruanda”).Why?SVD/PCAcomputationallyexpensiveHighercomplexityinimplementationNoclearexamplesofhigherperformancethroughdimensionreductionWordRep.vs.DimensionReduct21WordRep.vs.DimensionReductionWordRep.vs.DimensionReduct22MeasuringClassification

FiguresofMeritAccuracyofclassificationMainevaluationcriterioninacademiaMoreinamomenSpeedoftrainingstatisticalclassifierSpeedofclassification(docs/hour)NobigdifferencesformostalgorithmsExceptions:kNN,complexpreprocessingrequirementsEffortincreatingtrainingset(humanhours/topic)MoreonthisinLecture9(ActiveLearning)MeasuringClassification

Figur23MeasuresofAccuracyErrorrateNotagoodmeasureforsmallclasses.Why?Precision/recallforclassificationdecisionsF1measure:1/F1=½(1/P+1/R)BreakevenpointCorrectestimateofsizeofcategoryWhyisthisdifferent?Precision/recallforrankingclassesStabilityovertime/conceptdriftUtilityMeasuresofAccuracyErrorrate24Precision/RecallforRankingClassesExample:“BadwheatharvestinTurkey”TruecategoriesWheatTurkeyRankedcategorylist0.9:turkey0.7:poultry0.5:armenia0.4:barley0.3:georgiaPrecisionat5:0.1,Recallat5:0.5Precision/RecallforRankingC25Precision/RecallforRankingClassesConsiderproblemswithmanycategories(>10)Usemethodreturningscorescomparableacrosscategories(not:NaïveBayes)Rankcategoriesandcomputeaverageprecisionrecall(orothermeasurecharacterizingprecision/recallcurve)GoodmeasureforinteractivesupportofhumancategorizationUselessforan“autonomous”system(e.g.afilteronastreamofnewswirestories)Precision/RecallforRankingC26ConceptDriftCategorieschangeovertimeExample:“presidentoftheunitedstates”1999:clintonisgreatfeature2002:clintonisbadfeatureOnemeasureofatextclassificationsystemishowwellitprotectsagainstconceptdrift.Featureselection:goodorbadtoprotectagainstconceptdrift?ConceptDriftCategorieschange27Micro-vs.Macro-AveragingIfwehavemorethanoneclass,howdowecombinemultipleperformancemeasuresintoonequantity?Macroaveraging:Computeperformanceforeachclass,thenaverage.Microaveraging:Collectdecisionsforallclasses,computecontingencytable,evaluate.Micro-vs.Macro-AveragingIfw28Micro-vs.Macro-Averaging:ExampleTruth:yesTruth:noClassifier:yes1010Classifier:no10970Truth:yesTruth:noClassifier:yes9010Classifier:no10890Truth:yesTruth:noClassifier:yes10020Classifier:no201860Class1Class2Micro.Av.TableMacroaveragedprecision:(0.5+0.9)/2=0.7Microaveragedprecision:100/120=.83Whythisdifference?Micro-vs.Macro-Averaging:Ex29Reuters1NewswiretextStatistics(varyaccordingtoversionused)Trainingset:9,610Testset:3,66250%ofdocumentshavenocategoryassignedAveragedocumentlength:90.6Numberofclasses:92Exampleclasses:currencyexchange,wheat,goldMaxclassesassigned:14Averagenumberofclassesassigned1.24fordocswithatleastonecategoryReuters1Newswiretext30Reuters1Onlyabout10outof92categoriesarelargeMicroaveragingmeasuresperformanceonlargecategories.Reuters1Onlyabout10outof31FactorsAffectingMeasuresVariabilityofdataDocumentsize/lengthquality/styleofauthorshipuniformityofvocabularyVariabilityof“truth”/goldstandardneeddefinitivejudgementonwhichtopic(s)adocbelongstousuallyhumanIdeally:consistentjudgementsFactorsAffectingMeasuresVari32AccuracymeasurementConfusionmatrix53TopicassignedbyclassifierActualTopicThis(i,j)entrymeans53ofthedocsactuallyintopiciwereputintopicjbytheclassifier.AccuracymeasurementConfusion33ConfusionmatrixFunctionofclassifier,topicsandtestdocs.Foraperfectclassifier,alloff-diagonalentriesshouldbezero.Foraperfectclassifier,iftherearendocsincategoryjthanentry(j,j)shouldben.Straightforwardwhenthereis1categoryperdocument.Canbeextendedtoncategoriesperdocument.ConfusionmatrixFunctionofcl34Confusionmeasures(1class/doc)Recall:Fractionofdocsintopiciclassifiedcorrectly:Precision:Fractionofdocsassignedtopicithatareactuallyabouttopici:“Correctrate”:(1-errorrate)Fractionofdocsclassifiedcorrectly:Confusionmeasures(1class/35IntegratedEvaluation/OptimizationPrincipledapproachtotrainingOptimizethemeasurethatperformanceismeasuredwiths:vectorofclassifierdecision,z:vectoroftrueclassesh(s,z)=costofmakingdecisionssfortrueassignmentszIntegratedEvaluation/Optimiza36Utility/CostOnecostfunctionhisbasedoncontingencytable.Assumeidenticalcostforallfalsepositivesetc.CostC=l11*A+l12*B+l21*C+l22*DForthiscostc,wegetthefollowingoptimalitycriterionTruth:yesTruth:noClassifier:yesCost:λ11Count:ACost:λ12Count:BClassifier:noCost:λ21Count;CCost:λ22Count:DUtility/CostOnecostfunctio37Utility/CostTruth:yesTruth:noClassifier:yesλ11λ12Classifier:noλ21λ22Mostcommoncost:1forerror,0forcorrect.Pi>?Productcross-sale:highcostforfalsepositive,lowcostforfalsenegative.Patentsearch:lowcostforfalsepositive,highcostforfalsenegative.Utility/CostTruth:yesTruth:38

AreAllOptimalRulesofFormp>θ?Intheaboveexamples,allyouneedtodoisestimateprobabilityofclassmembership.Canallproblemsbesolvedlikethis?No!ProbabilityisoftennotsufficientUserdecisiondependsonthedistributionofrelevanceExample:informationfilterforterrorism

AreAllOptimalRulesofForm39NaïveBayesNaïveBayes40VectorSpaceClassification

NearestNeighborClassificationVectorSpaceClassification

Ne41RecallVectorSpaceRepresentationEachdocjisavector,onecomponentforeachterm(=word).Normalizetounitlength.Haveavectorspacetermsareaxesndocsliveinthisspaceevenwithstemming,mayhave10000+dimensions,oreven1,000,000+RecallVectorSpaceRepresenta42ClassificationUsingVectorSpacesEachtrainingdocapoint(vector)labeledbyitstopic(=class)Hypothesis:docsofthesametopicformacontiguousregionofspaceDefinesurfacestodelineatetopicsinspaceClassificationUsingVectorSp43TopicsinavectorspaceGovernmentScienceArtsTopicsinavectorspaceGovern44GivenatestdocFigureoutwhichregionitliesinAssigncorrespondingclassGivenatestdocFigureoutwhi45Testdoc=GovernmentGovernmentScienceArtsTestdoc=GovernmentGovernmen46BinaryClassificationConsider2classproblemsHowdowedefine(andfind)theseparatingsurface?Howdowetestwhichregionatestdocisin?BinaryClassificationConsider47SeparationbyHyperplanesAssumelinearseparabilityfornow:in2dimensions,canseparatebyalineinhigherdimensions,needhyperplanesCanfindseparatinghyperplanebylinearprogramming(e.g.perceptron):separatorcanbeexpressedasax+by=cSeparationbyHyperplanesAssum48Linearprogramming/PerceptronFinda,b,c,suchthatax+by

cforredpointsax+by

cforgreenpoints.Linearprogramming/Perceptro49RelationshiptoNaïveBayes?Finda,b,c,suchthatax+by

cforredpointsax+by

cforgreenpoints.RelationshiptoNaïveBayes?Fi50LinearClassifiersManycommontextclassifiersarelinearclassifiersDespitethissimilarity,largeperformancedifferencesForseparableproblems,thereisaninfinitenumberofseparatinghyperplanes.Whichonedoyouchoose?Whattodofornon-separableproblems?LinearClassifiersManycommon51Whichhyperplane?Ingeneral,lotsofpossiblesolutionsfora,b,c.Whichhyperplane?Ingeneral,l52SupportVectorMachine(SVM)SupportvectorsMaximizemarginQuadraticprogrammingproblem

Thedecisionfunctionisfullyspecifiedbysubsetoftrainingsamples,thesupportvectors.TextclassificationmethoddujourTopicoflecture9SupportVectorMachine(SVM)Su53Category:InterestExampleSVMfeatures

witiwiti0.70prime0.67rate0.63interest0.60rates0.46discount0.43bundesbank0.43baker-0.71dlrs-0.35world-0.33sees-0.25year-0.24group-0.24dlr-0.24januaryCategory:InterestExampleSVM54MoreThanTwoClassesAny-oformulticlassclassificationFornclasses,decomposeintonbinaryproblemsOne-ofclassification:eachdocumentbelongstoexactlyoneclassHowdowecomposeseparatingsurfacesintoregions?CentroidclassificationKnearestneighborclassificationMoreThanTwoClassesAny-ofor55ComposingSurfaces:Issues???ComposingSurfaces:Issues???56SeparatingMultipleTopicsBuildaseparatorbetweeneachtopicanditscomplementaryset(docsfromallothertopics).Giventestdoc,evaluateitformembershipineachtopic.DeclaremembershipintopicsOne-ofclassification:forclasswithmaximumscore/confidence/probabilityMulticlassclassification:ForclassesabovethresholdSeparatingMultipleTopicsBuil57NegativeexamplesFormulateasabove,exceptnegativeexamplesforatopicareaddedtoitscomplementaryset.PositiveexamplesNegativeexamplesNegativeexamplesFormulateas58CentroidClassificationGiventrainingdocsforatopic,computetheircentroidNowhaveacentroidforeachtopicGivenquerydoc,assigntotopicwhosecentroidisnearest.Exercise:ComparetoRocchioCentroidClassificationGivent59ExampleGovernmentScienceArtsExampleGovernmentScienceArts60kNearestNeighborClassificationToclassifydocumentdintoclasscDefinek-neighborhoodNasknearestneighborsofdCountnumberofdocumentslinNthatbelongtocEstimateP(c|d)asl/kkNearestNeighborClassificat61CoverandHart1967Asymptotically,theerrorrateof1-nearest-neighborclassificationislessthantwicetheBayesrate.Assumethatquerypointcoincideswithatrainingpoint.Bothquerypointandtrainingpointcontributeerror->2timesBayesrateCoverandHart1967Asymptotica62kNNvs.RegressionkNNhashighvarianceandlowbias.Linearregressionhaslowvarianceandhighbias.kNNvs.RegressionkNNhashigh63kNN:DiscussionClassificationtimelinearintrainingsetTrainingsetgenerationincompletelyjudgedsetcanbeproblematicformulticlassproblemsNofeatureselectionnecessaryScaleswellwithlargenumberofcategoriesDon’tneedtotrainnclassifiersfornclassesCategoriescaninfluenceeachotherSmallchangestoonecategorycanhaverippleeffectScorescanbehardtoconverttoprobabilitiesNotrainingnecessaryActually:nottrue.Why?kNN:DiscussionClassification64NumberofneighborsNumberofneighbors65ReferencesAComparativeStudyonFeatureSelectioninTextCategorization(1997)

YimingYang,JanO.Pedersen.ProceedingsofICML-97,14thInternationalConferenceonMachineLearning.EvaluatingandOptimizingAutonomousTextClassificationSystems(1995)

DavidLewis.Proceedingsofthe18thAnnualInternationalACMSIGIRConferenceonResearchandDevelopmentinInformationRetrievalFoundationsofStatisticalNaturalLanguageProcessing.Chapter16.MITPress.ManningandSchuetze.TrevorHastie,RobertTibshiraniandJeromeFriedman,"ElementsofStatisticalLearning:DataMining,InferenceandPrediction"Springer-Verlag,NewYork.ReferencesAComparativeStudy66KappaMeasureKappameasuresAgreementamongcodersDesignedforcategoricaljudgmentsCorrectsforchanceagreementKappa=[P(A)–P(E)]/[1–P(E)]P(A)–proportionoftimecodersagreeP(E)–whatagreementwouldbebychanceKappa=0forchanceagreement,1fortotalagreement.KappaMeasureKappameasures67CS276B

TextInformationRetrieval,Mining,andExploitationLecture523January2003CS276B

TextInformationRetrie68RecapRecap69Today’stopicsFeatureselectionfortextclassificationMeasuringclassificationperformanceNearestneighborcategorizationToday’stopicsFeatureselectio70FeatureSelection:Why?Textcollectionshavealargenumberoffeatures10,000–1,000,000uniquewords–andmoreMakeusingaparticularclassifierfeasibleSomeclassifierscan’tdealwith100,000soffeat’sReducetrainingtimeTrainingtimeforsomemethodsisquadraticorworseinthenumberoffeatures(e.g.,logisticregression)ImprovegeneralizationEliminatenoisefeaturesAvoidoverfittingFeatureSelection:Why?Textco71Recap:FeatureReductionStandardwaysofreducingfeaturespacefortextStemmingLaugh,laughs,laughing,laughed->laughStopwordremovalE.g.,eliminateallprepositionsConversiontolowercaseTokenizationBreakonallspecialcharacters:fire-fighter->fire,fighterRecap:FeatureReductionStanda72FeatureSelectionYangandPedersen1997ComparisonofdifferentselectioncriteriaDF–documentfrequencyIG–informationgainMI–mutualinformationCHI–chisquareCommonstrategyComputestatisticforeachtermKeepntermswithhighestvalueofthisstatisticFeatureSelectionYangandPede73InformationGainInformationGain74(Pointwise)MutualInformation(Pointwise)MutualInformation75Chi-SquareTermpresentTermabsentDocumentbelongstocategoryABDocumentdoesnotbelongtocategoryCDX^2=N(AD-BC)^2/((A+B)(A+C)(B+D)(C+D))UseeithermaximumoraverageX^2Valueforcompleteindependence?Chi-SquareTermpresentTermabs76DocumentFrequencyNumberofdocumentsatermoccursinIssometimesusedforeliminatingbothveryfrequentandveryinfrequenttermsHowisdocumentfrequencymeasuredifferentfromtheother3measures?DocumentFrequencyNumberofdo77Yang&Pedersen:ExperimentsTwoclassificationmethodskNN(knearestneighbors;morelater)LinearLeastSquaresFitRegressionmethodCollectionsReuters-2217392categories16,000uniquetermsOhsumed:subsetofmedline14,000categories72,000uniquetermsLtctermweightingYang&Pedersen:ExperimentsTwo78Yang&Pedersen:ExperimentsChoosefeaturesetsizePreprocesscollection,discardingnon-selectedfeatures/wordsApplytermweighting->featurevectorforeachdocumentTrainclassifierontrainingsetEvaluateclassifierontestsetYang&Pedersen:ExperimentsChoo79浅探关节镜下盘状半月板损伤的治疗课件80DiscussionYoucaneliminate90%offeaturesforIG,DF,andCHIwithoutdecreasingperformance.Infact,performanceincreaseswithfewerfeaturesforIG,DF,andCHI.Mutualinformationisverysensitivetosmallcounts.IGdoesbestwithsmallestnumberoffeatures.Documentfrequencyisclosetooptimal.Byfarthesimplestfeatureselectionmethod.SimilarresultsforLLSF(regression).DiscussionYoucaneliminate9081ResultsWhyisselectingcommontermsagoodstrategy?ResultsWhyisselectingcommon82IG,DF,CHIAreCorrelated.IG,DF,CHIAreCorrelated.83InformationGainvsMutualInformationInformationgainissimilartoMIforrandomvariablesIndependence?Incontrast,pointwiseMIignoresnon-occurrenceoftermsE.g.,forcompletedependence,youget:P(AB)/(P(A)P(B))=1/P(A)–largerforraretermsthanforfrequenttermsYang&Pedersen:PointwiseMIfavorsraretermsInformationGainvsMutualInf84FeatureSelection:

OtherConsiderationsGenericvsClass-SpecificCompletelygeneric(class-independent)SeparatefeaturesetforeachclassMixed(alaYang&Pedersen)MaintainabilityovertimeIsaggressivefeaturesselectiongoodorbadforrobustnessovertime?Ideal:OptimalfeaturesselectedaspartoftrainingFeatureSelection:

OtherConsi85Yang&Pedersen:LimitationsDon’tlookatclassspecificfeatureselectionDon’tlookatmethodsthatcan’thandlehigh-dimensionalspacesEvaluatecategoryranking(asopposedtoclassificationaccuracy)Yang&Pedersen:LimitationsDon’86FeatureSelection:OtherMethodsStepwisetermselectionForwardBackwardExpensive:needtodon^2iterationsoftrainingTermclusteringDimensionreduction:PCA/SVDFeatureSelection:OtherMetho87WordRep.vs.DimensionReductionWordrepresentations:onedimensionforeachword(binary,count,orweight)Dimensionreduction:eachdimensionisauniquelinearcombinationofallwords(linearcase)Dimensionreductionisgoodforgenerictopics(“politics”),badforspecificclasses(“ruanda”).Why?SVD/PCAcomputationallyexpensiveHighercomplexityinimplementationNoclearexamplesofhigherperformancethroughdimensionreductionWordRep.vs.DimensionReduct88WordRep.vs.DimensionReductionWordRep.vs.DimensionReduct89MeasuringClassification

FiguresofMeritAccuracyofclassificationMainevaluationcriterioninacademiaMoreinamomenSpeedoftrainingstatisticalclassifierSpeedofclassification(docs/hour)NobigdifferencesformostalgorithmsExceptions:kNN,complexpreprocessingrequirementsEffortincreatingtrainingset(humanhours/topic)MoreonthisinLecture9(ActiveLearning)MeasuringClassification

Figur90MeasuresofAccuracyErrorrateNotagoodmeasureforsmallclasses.Why?Precision/recallforclassificationdecisionsF1measure:1/F1=½(1/P+1/R)BreakevenpointCorrectestimateofsizeofcategoryWhyisthisdifferent?Precision/recallforrankingclassesStabilityovertime/conceptdriftUtilityMeasuresofAccuracyErrorrate91Precision/RecallforRankingClassesExample:“BadwheatharvestinTurkey”TruecategoriesWheatTurkeyRankedcategorylist0.9:turkey0.7:poultry0.5:armenia0.4:barley0.3:georgiaPrecisionat5:0.1,Recallat5:0.5Precision/RecallforRankingC92Precision/RecallforRankingClassesConsiderproblemswithmanycategories(>10)Usemethodreturningscorescomparableacrosscategories(not:NaïveBayes)Rankcategoriesandcomputeaverageprecisionrecall(orothermeasurecharacterizingprecision/recallcurve)GoodmeasureforinteractivesupportofhumancategorizationUselessforan“autonomous”system(e.g.afilteronastreamofnewswirestories)Precision/RecallforRankingC93ConceptDriftCategorieschangeovertimeExample:“presidentoftheunitedstates”1999:clintonisgreatfeature2002:clintonisbadfeatureOnemeasureofatextclassificationsystemishowwellitprotectsagainstconceptdrift.Featureselection:goodorbadtoprotectagainstconceptdrift?ConceptDriftCategorieschange94Micro-vs.Macro-AveragingIfwehavemorethanoneclass,howdowecombinemultipleperformancemeasuresintoonequantity?Macroaveraging:Computeperformanceforeachclass,thenaverage.Microaveraging:Collectdecisionsforallclasses,computecontingencytable,evaluate.Micro-vs.Macro-AveragingIfw95Micro-vs.Macro-Averaging:ExampleTruth:yesTruth:noClassifier:yes1010Classifier:no10970Truth:yesTruth:noClassifier:yes9010Classifier:no10890Truth:yesTruth:noClassifier:yes10020Classifier:no201860Class1Class2Micro.Av.TableMacroaveragedprecision:(0.5+0.9)/2=0.7Microaveragedprecision:100/120=.83Whythisdifference?Micro-vs.Macro-Averaging:Ex96Reuters1NewswiretextStatistics(varyaccordingtoversionused)Trainingset:9,610Testset:3,66250%ofdocumentshavenocategoryassignedAveragedocumentlength:90.6Numberofclasses:92Exampleclasses:currencyexchange,wheat,goldMaxclassesassigned:14Averagenumberofclassesassigned1.24fordocswithatleastonecategoryReuters1Newswiretext97Reuters1Onlyabout10outof92categoriesarelargeMicroaveragingmeasuresperformanceonlargecategories.Reuters1Onlyabout10outof98FactorsAffectingMeasuresVariabilityofdataDocumentsize/lengthquality/styleofauthorshipuniformityofvocabularyVariabilityof“truth”/goldstandardneeddefinitivejudgementonwhichtopic(s)adocbelongstousuallyhumanIdeally:consistentjudgementsFactorsAffectingMeasuresVari99AccuracymeasurementConfusionmatrix53TopicassignedbyclassifierActualTopicThis(i,j)entrymeans53ofthedocsactuallyintopiciwereputintopicjbytheclassifier.AccuracymeasurementConfusion100ConfusionmatrixFunctionofclassifier,topicsandtestdocs.Foraperfectclassifier,alloff-diagonalentriesshouldbezero.Foraperfectclassifier,iftherearendocsincategoryjthanentry(j,j)shouldben.Straightforwardwhenthereis1categoryperdocument.Canbeextendedtoncategoriesperdocument.ConfusionmatrixFunctionofcl101Confusionmeasures(1class/doc)Recall:Fractionofdocsintopiciclassifiedcorrectly:Precision:Fractionofdocsassignedtopicithatareactuallyabouttopici:“Correctrate”:(1-errorrate)Fractionofdocsclassifiedcorrectly:Confusionmeasures(1class/102IntegratedEvaluation/OptimizationPrincipledapproachtotrainingOptimizethemeasurethatperformanceismeasuredwiths:vectorofclassifierdecision,z:vectoroftrueclassesh(s,z)=costofmakingdecisionssfortrueassignmentszIntegratedEvaluation/Optimiza103Utility/CostOnecostfunctionhisbasedoncontingencytable.Assumeidenticalcostforallfalsepositivesetc.CostC=l11*A+l12*B+l21*C+l22*DForthiscostc,wegetthefollowingoptimalitycriterionTruth:yesTruth:noClassifier:yesCost:λ11Count:ACost:λ12Count:BClassifier:noCost:λ21Count;CCost:λ22Count:DUtility/CostOnecostfunctio104Utility/CostTruth:yesTruth:noClassifier:yesλ11λ12Classifier:noλ21λ22Mostcommoncost:1forerror,0forcorrect.Pi>?Productcross-sale:highcostforfalsepositive,lowcostforfalsenegative.Patentsearch:lowcostforfalsepositive,highcostforfalsenegative.Utility/CostTruth:yesTruth:105

AreAllOptimalRulesofFormp>θ?Intheaboveexamples,allyouneedtodoisestimateprobabilityofclassmembership.Canallproblemsbesolvedlikethis?No!ProbabilityisoftennotsufficientUserdecisiondependsonthedistributionofrelevanceExample:informationfilterforterrorism

AreAllOptimalRulesofForm106NaïveBayesNaïveBayes107VectorSpaceClassification

NearestNeighborClassificationVector