大数据如何帮助小数据-FeiSha南加州大学副教授

Howbigdatacanhelpsmalldata?DepartmentofComputerScienceUniversityofSouthernCaliforniaEdgecasestudy“常常会遇到偶发的⼤QuanFinance“左尾”／“右尾”没有⾜够多的数据TimescalesShort-termWhichschool?TimescalesShort-termWhichschool?Extracurricularactivities:sports,arts,etc?Mid-termWhichuniversity?Whichspeciality?Long-termWhatkindofcareerpath?Whatelsemakesthishard?IndividualdifferencesGeneticallyEnvironmentallyHarshconstraintsOne-shotgameCostlytorecoverfrommistakes[Bill&MelindaGatesFoundationonPersonalizedLearning]CNNonPersonalizedLearning6]CanyoubuildamodelofSadly,Sadly,not100%yetIndividualizedmodelsneedindividual-specificdataTheamountofdataisfundamentallylimited,hencebeingSmall.MostmodernlearningalgorithmsrequireBigDataabouttheindividual.Sadly,not100%yetIndividualizedmodelsneedindividual-specifiSadly,not100%yetIndividualizedmodelsneedindividual-specificdataTheamountofdataisfundamentallylimited,hencebeingSmall.MostmodernlearningalgorithmsrequireBigDataabouttheindividual.you.CanyoubuildamodelofSadly,Sadly,not100%yetIndividualizedmodelsneedindividual-specificdataTheamountofdataisfundamentallylimited,hencebeingSmall.MostmodernlearningalgorithmsrequireBigDataabouttheindividual.you.rI3LearningsettingsMulti-tasklearningDomainadaptationZero-shotlearningPrimaryapplicationfocusComputervision3LearningsettingsMulti-tasklearningDomainadaptationZero-shotlearningPrimaryapplicationfocusComputervision3LearningsettingsMulti-tasklearningDomainadaptationZero-shotlearningPrimaryapplicationfocusComputervision3LearningsettingsMulti-tasklearningDomainadaptationZero-shotlearningPrimaryapplicationfocusComputervision3LearningsettingsMulti-tasklearningDomainadaptationZero-shotlearningPrimaryapplicationfocusComputervisionVignette1Multi-taskLearning(MTL)“众人拾柴高”w1w2w3w4w1w2w3w4ProblemsettingMtasks,eachwithitsowndataNeedtofindsolutionsforallofthemTraditionalframeworkforsupervisedlearningSolveeachtaskindependentlyargmin`(Dm;wm)+λmR(wm)wmw1w2w1w2w3w4MainideaLearnjointlymultiplerelatedtasksForceknowledgesharingCombinesmalldataintobigdataBenefitsImprovegeneralizationperformanceRequirelessamountofdataWorksinbothdeepandshallowlearningmodelsw1,w2,···,wMMX`(Dm;wm)+λR(w1,w2,···,wM)m=1eeetalArgyriouetal08,Daumé,09..…]MX`(Dm;wm)+m=1MX`(Dm;wm)+m=1w1,w2,···,wMExploitingtaskrelatednessEncodepriorknowledgebyselectingtheregularizerConstrainthehypothesisspaceforalltasksChoicesofregularizerAllparametersaresimilartoeachotherParametersshouldhavesimilarsparsitypatterns.λλR(w1,w2,···,wM)ww1ww2ww3ww4D23InputvisualfeatureD[objectcategoriesandattributes,CVPR,2011]haredfeaturesxxxxD23InputvisualfeatureD[objectcategoriesandattributes,CVPR,2011]haredfeaturesxxxxwhitespotspolarbearwhitespotsject yaAAttributesclassifier classject yaAAttributesclassifier 911912 u1911912 u1u2u3MA11VisualfeaturespaceAnalogiesleopard:cat=wolf:dogleopard:tiger=horse:zebraRRegularization--------==SemanticEmbeddingSpace[Analogy-preservingembedding,ICML,2013]NBSharingontologiesNIPS2)]NotalltasksarebeneficialNotalltasksarebeneficialNotalltasksarebeneficialNotalltasksarebeneficialw1Howtodiscovergroupsofrelatedsubtasks?“Learningwithwhomtow1Howtodiscovergroupsofrelatedsubtasks?“Learningwithwhomtoshare”(ICML,2011)Group1w2“Resistingthetemptationtoshare”(CVPR,2014)Whythisisuseful?w3LearninginnoisytaskdataLearningfromasetofirrelevanttasksEx:compbio,noisylabelsGroup2w4NotalltasksVignette2DomainadaptationClassificationtask:givenafaceimage,determinemanorwoman?CollectalotoflabeledimagestrainingtaanwomanxxxxInferaclassificationboundary22 xxClassifyontestimagex2xxxClassifyontestimagex2xccessSharedstatisticalproperties,usefulforclassificationSharedstatisticalproperties,usefulforclassificationtell-talefeature:lengthofhairtrainingdatatestdataMismatchbetweentrainingandtestingtrainingdatatestdataunseendataMismatchbetweentrainingandtesting“lengthofhair”nolongerefective!trainingdatatestdataunseendataUnrealistic,oversimplifyingassumptionsLearningenvironmentisstationaryTraining,testingandfuturedataaresampledini.i.dfromthesamedistributionWorkswellinacademic/well-controlledsettings.Inreal-life,Learningenvironmentchanges.Training,testingandfuturedataaresampledfromdifferentdistributions.Wesufferfrompoorcross-distributiongeneralization,whereaccuracyfordisparatedomainsdropssignificantly.ComputervisionObjectrecognition:train&testondifferentdatasetsVehiclepedestrianavoidancesystems:train&testindifferentvehicular/cityenvironmentsNaturallanguageprocessingSyntacticparsing:trainonbusinessarticlesbutappliedtomedicaljournalsSpeechrecognition:trainonnativespeakersbutappliedtoaccentedvoicesChallengesManyexogenousfactorsaffectvisualappearances:pose,illumination,camera’squality,etc.Collectingdataunderallpossiblecombinationsofthosefactorsisexpensive.Labelingthosedataisevenmorecostly.CaltechCaltech-256mAmazonDSLRExampleimagesfrom4domainsinourempiricalstudiesAccuracyAccuracy[Anonymoussource,2014]EffectofusingbiggerdatasetsforadaptationlargersourceAmazonWebcamImageNetAdaptedAmazonAdaptedImageNetHowtoadapt?linearsubspacesDomain-invariantfeaturesTheoreticalmotivationExploitintrinsicstructuresLearnkernelsdiscriminativeclusteringGrasGrassmannmanifoldofsubspacesSourcedomainGeodesicflowcapturesdomain-invariantrepresentation(forvisualrecognition)Targetdomain(ICML13,NIPS13)[Ben-Davidetal’06,Blitzeretal’06,DaumeIII’07,Panetal,09,SharedrepresentationExistenceofa(latent)featurespaceThemarginalsofsourceandtargetsarethesame(orsimilar)inthisspaceExistasingleclassifierworkswellonbothdomainseT[h]<eS[h]+A(PS,PT)+infh2H[eT[h]+eS[h]]howwellahowwellasingleclassifiercandodistributionsaresimilarGrassmannmanifoldofsubspacesTargetdomainGeodesicflowcapturesGrassmannmanifoldofsubspacesTargetdomainGeodesicflowcapturesdomain-invariantSourcedomainrepresentation(forvisualrecognition)PRDomain-invariantfeaturesParameterizedaslinearkernelmappingoforiginalfeaturesConstructedtominimizediscrepancybetweentwodomainsModeldomainswithsubspacesComputediscrepancyasdifferencesbetweensubspacesGG(d,D)Noadaptation SGF(Gopalanetal,ICCV2011)GeodesicFlowkernel(ours)DAC45004500Geodesicflowkernel(GFK)LandmarkC-->AA-->WW-->CD-->AC-->DA-->CVignette3Zero-shotlearningClassicalmachinelearningframeworkMultiwayclassificationLabelingspaceisdeterminedapriorAlargenumberofannotatedtrainingsamplesforeveryclassChallengesforrecognitioninthewildLabelingspacegrowsarbitrarilylargewithemergenceofnewclassesCollectingdatafornewclassesisnotalwayscost-effectiveSomeclassesdonothaveenoughlabeledorzerolabeledimages“cat”“flower”“bench”“dog”“bear”“bird”Numberofspecies(total:1,589,361)Birds:9956Fish:30,000Mammals:5,416Reptiles:8,240Insects:950,000Corals:2,175Plants:297,326Mushrooms:16,000“Skywalker”gibbonObjectsSimilarly,inImageNetTwotypesofclassesSeen:withalotoflabeledexamplesUnseen:withoutanyexamplesCatHorse ?FiguresfromDerekHoiem’sslidesWhatisit:bear-like,withblackandwhitestripeandoftenwithbamboo?ClasslabelsClasslabels≠discretenumbersNeedtoassignsemanticmeaningstoclasslabelsNeedtodefinerelationshipsamongclasslabelsKeyassumptionsThereisacommonsemanticspacesharedbybothtypesofclassesConfigurationoftheembeddingsenable“transfer”.seeseenclassuneenclassSemanticEmbeddings•Attributes(Farhadietal.09,Lampertetal.09,Parikh&Grauman11,…)•Wordvectors(Mikolovetal.13,Socheretal.13,Fromeetal.13,…)•Word•Wordvectors(Mikolovetal.13,Socheretal.13,Fromeetal.13,…)SemanticEmbeddings•Attributes(Farhadietal.09,Lampertetal.09,Parikh&Grauman11,…)ngSeenObjectsnObjectSeenObjectsnObjectBrownMuscularHasSnoutHasMane(likehorse)HasSnout(likedog)HowHowtoeffectivelyconstructamodelforzebra?FiguresfromDerekHoiem’sslidesTrainingSeenclassesandtheirsemanticembeddingsS={1,2,···,S}AS={a1,a2,···,aS}AnnotatedtrainingsamplesD={(xn,yn)}=1GoalUnseenclassesandtheirsemanticembeddings八={S+1,···,S+U}AU={aS+1,aS+2,···,aS+U}Classifier:f:x!y2八ardaCardinal2v2w3311ModelspacewvvwardaCardinal2v2w3311Modelspacewvvwb1a1b2SemanticspaceSemanticspaceb3aSynthesizedclassifiersforzero-shotlearningSemanticrepresentationsSemanticembeddingspaceVisualfeaturesaGadwallaCedarWaxwinga(·)=PCAaHouseWren((au)forNNclassificationortoimproveexistingZSLapproaches:classexemplarcat01.11b1a1penguin−.2cat01.11b1a1penguin−.2Modelspace2−1.0(0.4A(−0.3Av1v2b2a3 Semanticspace3abBBC(−0.4AIntroducephantomclassesasbasesLearnbases’semanticembeddingsaswellasmodelsforbasesGraphsstructuresencode“relatedness”DefinehowclassesarerelatedinthesemanticembeddingspaceDefinehowclassesarerelatedinthemodelspaceDatasetsDatasetsTotal#AwA†CUB‡ClassificationaccuracyAwACUBSUNImageNet