迁移学习算法研究-庄福振New

TrainingDataClassifierUnseenData(…,long,T)good!Whatif…1传统监督机器学习(1/2)2023/4/12[fromProf.QiangYang]第一页，共86页。传统监督机器学习(2/2)22023/4/12传统监督学习同源、独立同分布两个基本假设标注足够多的训练样本在实际应用中通常不能满足！训练集测试集分类器训练集测试集分类器第二页，共86页。迁移学习32023/4/12实际应用学习场景HP新闻Lenovo新闻不同源、分布不一致人工标记训练样本，费时耗力迁移学习运用已有的知识对不同但相关领域问题进行求解的一种新的机器学习方法放宽了传统机器学习的两个基本假设第三页，共86页。迁移学习场景(1/4)42023/4/12迁移学习场景无处不在迁移知识迁移知识图像分类HP新闻Lenovo新闻新闻网页分类第四页，共86页。异构特征空间5Theappleisthepomaceousfruitoftheappletree,speciesMalusdomesticaintherosefamilyRosaceae...BananaisthecommonnameforatypeoffruitandalsotheherbaceousplantsofthegenusMusawhichproducethiscommonlyeatenfruit...Training:TextFuture:ImagesApplesBananas迁移学习场景(2/4)2023/4/12[fromProf.QiangYang]XinJin,FuzhenZhuang,SinnoJialinPan,ChangyingDu,PingLuo,QingHe:HeterogeneousMulti-taskSemanticFeatureLearningforClassification.CIKM2015:1847-1850.第五页，共86页。TestTestTrainingTrainingClassifierClassifier72.65%DVDElectronicsElectronics84.60%ElectronicsDrop!迁移学习场景(3/4)62023/4/12[fromProf.QiangYang]第六页，共86页。7DVDElectronicsBookKitchenClothesVideogameFruitHotelTeaImpractical!迁移学习场景(4/4)2023/4/12[fromProf.QiangYang]第七页，共86页。OutlineConceptLearningforTransferLearningConceptLearningbasedonNon-negativeMatrixTri-factorizationforTransferLearningConceptLearningbasedonProbabilisticLatentSemanticAnalysisforTransferLearningTransferLearningusingAuto-encodersTransferLearningfromMultipleSourceswithAutoencoderRegularizationSupervisedRepresentationLearning:TransferLearningwithDeepAuto-encoders82023/4/12第八页，共86页。ConceptLearningbasedonNon-negativeMatrixTri-factorizationforTransferLearningConceptLearningforTransferLearning92023/4/12第九页，共86页。IntroductionManytraditionallearningtechniquesworkwellonlyundertheassumption:Trainingandtestdatafollowthesamedistribution

Training(labeled)ClassifierTest(unlabeled)EnterpriseNewsClassification:includingtheclasses“ProductAnnouncement”,“Businessscandal”,“Acquisition”,……Productannouncement:HP'sjust-releasedLaserJetProP1100printerandtheLaserJetProM1130andM1210multifunctionprinters,price…performance

...AnnouncementforLenovoThinkPad

ThinkCentre–price$150offLenovoK300desktopusingcouponcode...LenovoThinkPad

ThinkCentre–price$200offLenovoIdeaPadU450plaptopusing....theirperformanceHPnewsLenovonewsDifferentdistributionFail!10ConceptLearningforTransferLearning2023/4/12第十页，共86页。Motivation(1/3)ExampleAnalysis

Productannouncement:HP'sjust-releasedLaserJetProP1100printerandtheLaserJetProM1130andM1210multifunctionprinters,price…performance

...AnnouncementforLenovoThinkPad

ThinkCentre–price$150offLenovoK300desktopusingcouponcode...LenovoThinkPad

ThinkCentre–price$200offLenovoIdeaPadU450plaptopusing....theirperformanceHPnewsLenovonewsProductwordconceptLaserJet,printer,price,performanceThinkPad,ThinkCentre,price,performanceRelatedProductannouncementdocumentclass:11Sharesomecommonwords:announcement,price,performance…indicateConceptLearningforTransferLearning2023/4/12第十一页，共86页。Motivation(2/3)ExampleAnalysis:

HPLaserJet,printer,price,performanceetal.LenovoThinkpad,Thinkcentre,price,performanceetal.Thewordsexpressingthesamewordconceptaredomain-dependent

12ProductProductannouncementwordconceptindicatesTheassociationbetweenwordconceptsanddocumentclassesisdomain-independent

ConceptLearningforTransferLearning2023/4/12第十二页，共86页。Motivation(3/3)13Furtherobservations:Differentdomainsmayusesamekeywordstoexpressthesameconcept(denotedasidenticalconcept)Differentdomainsmayalsousedifferentkeywordstoexpressthesameconcept(denotedasalikeconcept)Differentdomainsmayalsohavetheirowndistinctconcepts(denotedasdistinctconcept)TheidenticalandalikeconceptsareusedasthesharedconceptsforknowledgetransferWetrytomodelthesethreekindsofconceptssimultaneouslyfortransferlearningtextclassificationConceptLearningforTransferLearning2023/4/12第十三页，共86页。PreliminaryKnowledgeBasicformulaofmatrixtri-factorization:wheretheinputXistheword-documentco-occurrencematrix

FGS14ConceptLearningforTransferLearning2023/4/12第十四页，共86页。Previousmethod-MTrickinSDM2010(1/2)SketchmapofMTrick

SourcedomainXs

FsGsFtGtTargetdomainXtSKnowledgeTransfer15ConceptLearningforTransferLearning2023/4/12Consideringthealikeconcepts 第十五页，共86页。MTrick(2/2)OptimizationproblemforMTrickG0isthesupervisioninformationtheassociationSissharedasbridgetotransferknowledge16ConceptLearningforTransferLearningDualTransferLearning(Longetal.,SDM2012),consideringidenticalandalikeconcepts2023/4/12第十六页，共86页。TriplexTransferLearning(TriTL)(1/5)Furtherdividethewordconceptsintothreekinds:

17F1,identicalconcepts;F2,alikeconcepts;F3,distinctconceptsInput:ssourcedomainXr(1≤r≤s)withlabelinformation,ttargetdomainXr(s+1≤r≤s+t)WeproposeTriplexTransferLearningframeworkbasedonmatrixtri-factorization(TriTLforshort)

2023/4/12ConceptLearningforTransferLearning第十七页，共86页。F1,S1andS2

aresharedasthebridgeforknowledgetransferacrossdomainsThesupervisioninformationisintegratedbyGr(1≤r≤s)insourcedomainsTriTL(2/5)OptimizationProblem

182023/4/12ConceptLearningforTransferLearning第十八页，共86页。TriTL(3/5)Wedevelopanalternativelyiterativealgorithmtoderivethesolutionandtheoreticallyanalyzeitsconvergence 192023/4/12ConceptLearningforTransferLearning第十九页，共86页。TriTL(4/5)Classificationontargetdomains When1≤r≤s,Grcontainsthelabelinformation,soweremainitunchangedduringtheiterationswhenxibelongstoclassj,thenGr(i,j)=1,elseGr(i,j)=0Aftertheiteration,weobtaintheoutputGr(s+1≤r≤s+t),thenwecanperformclassificationaccordingtoGr202023/4/12ConceptLearningforTransferLearning第二十页，共86页。TriTL(5/5)AnalysisofAlgorithmConvergence Accordingtothemethodologyofconvergenceanalysisinthetwoworks[Leeetal.,NIPS’01]and[Dingetal.,KDD’06],thefollowingtheoremholds.Theorem(Convergence):Aftereachroundofcalculatingtheiterativeformulas,theobjectivefunctionintheoptimizationproblemwillconvergemonotonically.212023/4/12ConceptLearningforTransferLearning第二十一页，共86页。222023/4/12rec.autosrec.motorcyclesrec.baseballrec.hockeysci.cryptsic.electronicssci.medsci.spacecomp.graphicscomp.sys.ibm.pc.hardwarecomp.sys.mac.hardwarecomp.windows.xtalk.politics.misctalk.politics.gunstalk.politics.mideasttalk.religion.miscrecscicomptalkDataPreparation(1/3)20Newsgroups Fourtopcategories,eachtopcategorycontainsfoursub-categories SentimentClassification,fourdomains:books,dvd,electronics,kitchenRandomlyselecttwodomainsassources,andtherestastargets,then6problemscanbeconstructed ConceptLearningforTransferLearning第二十二页，共86页。232023/4/12rec.autosrec.motorcyclesrec.baseballrec.hockeysci.cryptsic.electronicssci.medsci.spacerec+sci-baseballcrypySourcedomainautosspaceTargetdomainFortheclassificationproblemwithonesourcedomainandonetargetdomain,wecanconstruct144()

problemsDataPreparation(2/3)Constructclassificationtasks(TraditionalTL)ConceptLearningforTransferLearning第二十三页，共86页。242023/4/12Constructnewtransferlearningproblemsrec.autosrec.motorcyclesrec.baseballrec.hockeysci.cryptsic.electronicssci.medsci.spacerec+sci-baseballcrypyautosspacecomp.graphicscomp.sys.ibm.pc.hardwarecomp.sys.mac.hardwarecomp.windows.xtalk.politics.misctalk.politics.gunstalk.politics.mideasttalk.religion.misccomptalkautosgraphicsMoredistinctconceptsmayexist!DataPreparation(3/3)SourcedomainTargetdomainConceptLearningforTransferLearning第二十四页，共86页。252023/4/12ComparedAlgorithmsConceptLearningforTransferLearningTraditionallearningAlgorithmsSupervisedLearning:LogisticRegression(LR)[Davidetal.,00]SupportVectorMachine(SVM)[Joachims,ICML’99]Semi-supervisedLearning:TSVM[Joachims,ICML’99]TransferlearningMethods:CoCC[Daietal.,KDD’07],DTL[Longetal.,SDM’12]Classificationaccuracyisusedastheevaluationmeasure第二十五页，共86页。262023/4/12ExperimentalResults(1/3)ConceptLearningforTransferLearningSorttheproblemswiththeaccuracyofLRDegreeoftransferdifficultyeasierGenerally,thelowerofaccuracyofLRcanindicatethehardertotransfer,whilethehigheronesindicatetheeasiertotransferharder第二十六页，共86页。272023/4/12ExperimentalResults(2/3)ConceptLearningforTransferLearningComparisonsamongTriTL,DTL,MTrick,CoCC,TSVM,SVMandLRondatasetrecvs.sci(144problems)TriTLcanperformwelleventheaccuracyofLRislowerthan65%第二十七页，共86页。282023/4/12ExperimentalResults(3/3)ConceptLearningforTransferLearningResultsonnewtransferlearningproblems,weonlyselecttheproblems,whoseaccuraciesofLRarebetween(50%,55%](Onlyslightlybetterthanrandomclassification,thustheymightbemuchmoredifficult).Weobtain65problemsTriTLalsooutperformsallthebaselines第二十八页，共86页。ConclusionsExplicitlydefinethreekindsofwordconcepts,i.e.,identicalconcept,alikeconceptanddistinctconceptProposeageneraltransferlearningframeworkbasedonnonnegativematrixtri-factorization,whichsimultaneouslymodelthethreekindsofconcepts(TriTL)Extensiveexperimentsshowtheeffectivenessoftheproposedapproach,especiallywhenthedistinctconceptsmayexist292023/4/12ConceptLearningforTransferLearning第二十九页，共86页。ConceptLearningbasedonProbabilisticLatentSemanticAnalysisforTransferLearningConceptLearningforTransferLearning302023/4/12第三十页，共86页。312023/4/12MotivationConceptLearningforTransferLearningProductannouncement:HP'sjust-releasedLaserJetProP1100printerandtheLaserJetProM1130andM1210multifunctionprinters,price…performance

...AnnouncementforLenovoThinkPad

ThinkCentre–price$150offLenovoK300desktopusingcouponcode...LenovoThinkPad

ThinkCentre–price$200offLenovoIdeaPadU450plaptopusing....theirperformanceHPnewsLenovonewsProductwordconceptLaserJet,printer,price,performanceThinkPad,ThinkCentre,price,performanceRelatedProductannouncementdocumentclass:Sharesomecommonwords:announcement,price,performance…indicateRetrospecttheexample

第三十一页，共86页。322023/4/12SomenotationsddocumentydocumentclasszwordconceptSomedefinitionse.g.,p(price|Product),p(LaserJet|Product,)wwordrdomaine.g,p(Product|Productannouncement)PreliminaryKnowledge(1/3)ConceptLearningforTransferLearning第三十二页，共86页。332023/4/12ConceptLearningforTransferLearningPreliminaryKnowledge(2/3)ProductLaserJet,printer,announcement,price,ThinkPad,ThinkCentre,announcement,priceProductannouncementp(w|z,r1)p(w|z,r2)p(z|y)p(w|z,r1)≠p(w|z,r2)E.g.,p(LaserJet|Product,HP)≠p(LaserJet|Product,Lenovo)p(z|y,r1)=p(z|y,r2)E.g.,p(Product|Productannoucement,HP)=p(Product|Productannoucement,Lenovo)Alikeconcept第三十三页，共86页。342023/4/12DualPLSA

(D-PLSA)Jointprobabilityoverallvariablesp(w,d)=p(w|z)p(z|y)p(d|y)p(y)GivendatadomainX,theproblemofmaximumloglikelihoodislogp(X;θ)=logΣz

p(Z,X;θ)

θ

includesalltheparametersp(w|z),p(z|y),p(d|y),p(y).Z

denotesallthelatentvariablesPreliminaryKnowledge(3/3)TheproposedtransferlearningalgorithmbasedonD-PLSA,denotedasHIDCConceptLearningforTransferLearning第三十四页，共86页。352023/4/12Identicalconceptp(w|za)p(za|y)AlikeconceptTheextensionandintensionaredomainindependentp(w|zb,r)p(zb|y)HIDC(1/3)Theextensionisdomaindependent,whiletheintensionisdomainindependentConceptLearningforTransferLearning第三十五页，共86页。362023/4/12Distinctconceptp(w|zc,r)p(zc|y,r)ThejointprobabilitiesofthesethreegraphicalmodelsHIDC(2/3)TheextensionandintensionarebothdomaindependentConceptLearningforTransferLearning第三十六页，共86页。372023/4/12Givens+t

datadomainsX={X1,…,Xs,Xs+1,…,Xs+t},withoutlossofgenerality,thefirstsdomainsaresourcedomains,andthelefttdomainsaretargetdomainsConsiderthethreekindsofconcepts:TheLog

likelihoodfunctionislogp(X;θ)=logΣz

p(Z,X;θ)

θ

includesallparametersp(w|za),p(w|zb,r),p(w|zc,r),p(za|y),p(zb|y),p(zc|y,r),p(d|y,r),p(y|r),p(r).HIDC(3/3)ConceptLearningforTransferLearning第三十七页，共86页。382023/4/12UsetheEMalgorithmtoderivethesolutionsEStep:ModelSolution(1/4)ConceptLearningforTransferLearning第三十八页，共86页。392023/4/12M

Step：ModelSolution(2/4)ConceptLearningforTransferLearning第三十九页，共86页。402023/4/12Semi-supervisedEMalgorithm:whenrisfromsourcedomains,thelabeledinformationp(d|y,r)isknownandp(y|r)

canbeinferedp(d|y,r)=1/ny,r,ifdbelongsyindomainr,ny,risthenumberofdocumentsinclassyindomainr,else

p(d|y,c)=0p(y|r)=ny,r/nr

,nr

isthenumberofdocumentsindomainr

whenrisfromsourcedomains,p(d|y,r)andp(y|r)keepunchangedduringtheiterations,whichsupervisetheoptimizingprocessModelSolution(3/4)ConceptLearningforTransferLearning第四十页，共86页。412023/4/12ClassificationfortargetdomainsAfterweobtainthefinalsolutionsofp(w|za),p(w|zb,r),p(w|zc,r),p(za|y),p(zb|y),p(zc|y,r),p(d|y,r),p(y|r),p(r)Wecancomputetheconditionalprobabilities:

ThenthefinalpredictionisDuringtheiterations,alldomainssharep(w|za),p(za|y),p(zb|y),

whichactasthebridgeforknowledgetransferModelSolution(4/4)ConceptLearningforTransferLearning第四十一页，共86页。422023/4/12BaselinesComparedAlgorithmsSupervisedLearning:LogisticRegression(LG)[Davidetal.,00]SupportVectorMachine(SVM)[Joachims,ICML’99]Semi-supervisedLearning:TSVM[Joachims,ICML’99]TransferLearning:CoCC[Daietal.,KDD’07]CD-PLSA[Zhuangetal.,CIKM’10]DTL[Longetal.,SDM’12]OurMethodsHIDCMeasure:classificationaccuracyConceptLearningforTransferLearning第四十二页，共86页。432023/4/12Resultsonnewtransferlearningproblems,weselecttheproblems,whoseaccuraciesofLRarehigherthan50%,then334problemsareobtainedExperimentalResults(1/5)ConceptLearningforTransferLearning第四十三页，共86页。442023/4/12Resultsonnewtransferlearningproblems,weselecttheproblems,whoseaccuraciesofLRarehigherthan50%,then334problemsareobtainedExperimentalResults(2/5)ConceptLearningforTransferLearning第四十四页，共86页。452023/4/12ExperimentalResults(3/5)ConceptLearningforTransferLearning第四十五页，共86页。462023/4/12Sourcedomain:S

(rec.autos,

sci.space)，Targetdomain:T(,)STSTDistinctconceptSTAlikeconceptExperimentalResults(4/5)ConceptLearningforTransferLearning第四十六页，共86页。472023/4/12ExperimentalResults(5/5)ConceptLearningforTransferLearningIndeed,theproposedprobabilisticmethodHIDCisalsobetterthanTriTLThismayduetothereasonthatthereismoreclearerprobabilisticexplanationofHIDCp1(z,y)=p2(z,y)orp1(z|y)=p2(z|y)whichisbetter?p(z|y)p(y)第四十七页，共86页。482023/4/12[1]FuzhenZhuang,PingLuo,HuiXiong,QingHe,YuhongXiong,ZhongzhiShi:ExploitingAssociationsbetweenWordClustersandDocumentClassesforCross-DomainTextCategorization.SDM2010,pp.13-24.[2]FuzhenZhuang,PingLuo,ZhiyongShen,QingHe,YuhongXiong,ZhongzhiShi,HuiXiong:CollaborativeDual-PLSA:miningdistinctionandcommonalityacrossmultipledomainsfortextclassification.CIKM2010,pp.359-368.[3]FuzhenZhuang,PingLuo,ZhiyongShen,QingHe,YuhongXiong,ZhongzhiShi,HuiXiong:MiningDistinctionandCommonalityacrossMultipleDomainsUsingGenerativeModelforTextClassification.IEEETrans.Knowl.DataEng.24(11):2025-2039(2012).[3]FuzhenZhuang,PingLuo,ChangyingDu,QingHe,ZhongzhiShi:Triplextransferlearning:exploitingbothsharedanddistinctconceptsfortextclassification.WSDM2013,pp.425-434.[4]FuzhenZhuang,PingLuo,PeifengYin,QingHe,ZhongzhiShi.:ConceptLearningforCross-domainTextClassification:aGeneralProbabilisticFramework.IJCAI2013,pp.1960-1966.ReferencesConceptLearningforTransferLearning第四十八页，共86页。OutlineConceptLearningforTransferLearningConceptLearningbasedonNon-negativeMatrixTri-factorizationforTransferLearningConceptLearningbasedonProbabilisticLatentSemanticAnalysisforTransferLearningTransferLearningusingAuto-encodersTransferLearningfromMultipleSourceswithAutoencoderRegularizationSupervisedRepresentationLearning:TransferLearningwithDeepAuto-encoders492023/4/12第四十九页，共86页。TransferLearningfromMultipleSourceswithAutoencoderRegularization502023/4/12TransferLearningUsingAuto-encoders第五十页，共86页。51Motivation(1/2)TransferlearningbasedonoriginalfeaturespacemayfailtoachievehighperformanceonTargetdomaindataWeconsidertheautoencodertechniquetocollaborativelyfindanewrepresentationofbothsourceandtargetdomaindataElectronicsVideoGames

Compact;easytooperate;verygoodpicture,excited

aboutthequality;lookssharp!Averygood

game!Itisactionpacked

andfullofexcitement.Iamverymuchhooked

onthisgame.512023/4/12TransferLearningUsingAuto-encoders第五十一页，共86页。PreviousmethodsoftentransferfromonesourcedomaintoonetargetdomainWeconsidertheconsensusregularizedframeworkforlearningfrommultiplesourcedomainsDVDBookKitchenElectronicsWeproposeatransferlearningframeworkofconsensusregularizationautoencoderstolearnfrommultiplesourcesMotivation(2/2)522023/4/12TransferLearningUsingAuto-encoders第五十二页，共86页。AutoencoderNeuralNetworkMinimizingthereconstructionerrortoderivethesolution:whereh,garenonlinearactivationfunction,e.g.,Sigmoidfunction,forencodinganddecoding532023/4/12TransferLearningUsingAuto-encoders第五十三页，共86页。ConsensusMeasure-(1/3)Example:three-classclassificationproblem,threeclassifierspredictinstancesf1f2f3f1f2f3x1111x2333x3222x4231x5313x6123ConstraintSource1:D1Source2:D2Source3:D3542023/4/12TransferLearningUsingAuto-encoders第五十四页，共86页。ConsensusMeasure-(2/3)Example:three-classclassificationproblem,predictiononinstancexMinimalentropy,MaximalConsensusMaximalentropy,MinimalConsensusEntropybasedConsensusMeasure(Luoetal.,CIKM’08)θiistheparametervectorofclassifieri,Cistheclasslabelset552023/4/12TransferLearningUsingAuto-encoders第五十五页，共86页。ConsensusMeasure-(3/3)Forsimplicity,theconsensusmeasureforbinaryclassificationcanberewrittenasInthiswork,weimposetheconsensusregularizationtoautoencoders,andtrytoimprovethelearningperformancefrommultiplesourcedomainssincetheireffectsonmakingthepredictionconsensusaresimilar.562023/4/12TransferLearningUsingAuto-encoders第五十六页，共86页。SomeNotationsSourcedomainsGivenrsourcedomains:,i.e.,

,.ThefirstcorrespondingdatamatrixisTargetdomainThecorrespondingdatamatrixisThegoalistotrainaclassifier

ftomakeprecisepredictionson.572023/4/12TransferLearningUsingAuto-encoders第五十七页，共86页。FrameworkofCRAThedatafromallsourceandtargetdomainssharethesameencodinganddecodingweightsTheclassifierstrainedfromthenewrepresentationareregularizedtopredictthesameresultsontargetdomaindata582023/4/12TransferLearningUsingAuto-encoders第五十八页，共86页。OptimizationProblemofCRATheoptimizationproblem:ReconstructionError592023/4/12TransferLearningUsingAuto-encoders第五十九页，共86页。OptimizationProblemofCRATheoptimizationproblem:ConsensusRegularization602023/4/12TransferLearningUsingAuto-encoders第六十页，共86页。OptimizationProblemofCRATheoptimizationproblem:ThetotallossofsourceclassifiersoverthecorrespondingsourcedomaindatawiththehiddenrepresentationWeighdecayterm612023/4/12TransferLearningUsingAuto-encoders第六十一页，共86页。TheSolutionofCRAWeusethegradientdescentmethodtoderivethesolutionofallparametersƞisthelearningrate.ThetimecomplexityisO(rnmk)Theoutput:theencodinganddecodingparameters,andsourceclassifierswithlatentrepresentation.622023/4/12TransferLearningUsingAuto-encoders第六十二页，共86页。TargetClassifierConstructionTwoScheme:Trainthesourceclassifiersbasedonandcombinethemas,whereCombineallthesourcedomaindataasZSandtrainaunifiedclassifierusinganysupervisedlearningalgorithms,e.g.,SVM,LogisticRegression(LR).ThetwoaccuraciesaredenotedasCRAvandCRAu,respectively632023/4/12TransferLearningUsingAuto-encoders第六十三页，共86页。DataSets-(1/2)ImageData(fromLuoetal.,CIKM08)(Someexamples)AB

A1A2A3A4B1B2B3B4Threesources:A1B1A2B2A3B3Targetdomain:A4B4Totally,96()3-sourcevs1-targetdomain(3vs1)probleminstancescanbeconstructedfortheexperimentalevaluation642023/4/12TransferLearningUsingAuto-encoders第六十四页，共86页。DataSets-(2/2)SentimentClassification(fromBlitzeretal.,ACL07)Four3-sourcevs1-targetdomainclassificationproblemsareconstructedDVDBookKitchenElectronicsTheaccuracyontargetdomaindataisusedastheevaluationmeasureBothSVMandLRareusedtotrainclassifiersonthenewrepresentation652023/4/12TransferLearningUsingAuto-encoders第六十五页，共86页。AllComparedAlgorithmsBaselinesSupervisedlearningonoriginalfeatures:SVM[Joachims,ICML’99],LogisticRegression(LR)[Davidetal.,00]Embeddingmethodbasedonautoencoders(EAER)[Yuetal.,ECML’13]MarginalizedStackedDenoisingAutoencoders

(mSDA)[Chenetal.,ICML’12]TransferComponentAnalysis(TCA)[Panetal.,TNN’11]Transferlearningfrommultiplesources(CCR3)(Luoetal.,CIKM’08)Ourmethod:CRAvandCRAuForthemethodswhichcannothandlemultiplesources,wetraintheclassifiersfromeachsourcedomainandmergeddataofallsources(r+1accuracies).Finally,maximal,meanandminimalvaluesarereported.662023/4/12TransferLearningUsingAuto-encoders第六十六页，共86页。67ExperimentalResults-(1/2)TransferLearningwithMultipleSourcesviaConsensusRegularizationAutoencodersFuzhenZhuang,XiaohuCheng,SinnoJialinPan,WenchaoYu,QingHe,andZhongzhiShiResultson96imageclassificationproblems第六十七页，共86页。68ExperimentalResults-(2/2)TransferLearningwithMultipleSourcesviaConsensusRegularizationAutoencodersFuzhenZhuang,XiaohuCheng,SinnoJialinPan,WenchaoYu,QingHe,andZhongzhiShiResultson4sentimentclassificationproblems第六十八页，共86页。ConclusionsThewellknownrepresentationlearningtechniqueautoencoderisconsidered,andweformalizetheautoencodersandconsensusregularizationintoaunifiedoptimizationframeworkExtensivecomparisonexperimentsonimageandsentimentdataareconductedtoshowtheeffectivenessoftheproposealgorithm692023/4/12TransferLearningUsingAuto-encoders第六十九页，共86页。SupervisedRepresentationLearning:TransferLearningwithDeepAuto-encoders702023/4/12TransferLearningUsingAuto-encoders第七十页，共86页。Autoencoderisanunsupervisedfeaturelearningalgorithm,whichcannoteffectivelymakeuseofthelabelinformationLimitationofBasicAutoencoderContributionofThisWorkWeextendAutoencodertomulti-layerstructure,andincorporatethelabelasonelayerMotivation712023/4/12TransferLearningUsingAuto-encoders第七十一页，共86页。源领域和目标领域共享编码和解码权重利用KL距离对隐层空间进行约束利用多类回归模型对类标层进行约束FrameworkofTLDA(1/5)722023/4/12TransferLearningUsingAuto-encoders第七十二页，共86页。目标是最小化重构误差：DeepAutoencoderFrameworkofTLDA(2/5)732023/4/12TransferLearningUsingAuto-encoders第七十三页，共86页。KL距离KL距离衡量的是两个概率分布的差异情况，计算公式如下：以上KL距离并不满足传统距离的对称性，在分类问题中，我们一般采用对称的KL距离，表示如下：Framework