解析元学习：了解功能少量射击任务的表示法 Unraveling Meta-Learning - Understanding Feature Representations for Few-Shot Tasks

UnravelingMeta-Learning:UnderstandingFeature

RepresentationsforFew-ShotTasks

HarichandanaVejendla

(50478049)

1

2

Definitions

•Meta-Learning:Meta-learningdescribesmachinelearningalgorithmsthatacquireknowledgeandunderstandingfromtheoutcomeofothermachinelearningalgorithms.Theylearnhowtobest

combinethepredictionsfromothermachine-learningalgorithms.

•Few-shotLearning:Few-ShotLearningisaMachineLearningframeworkthatenablesapre-trainedmodeltogeneralizeovernewcategoriesofdatausingonlyafewlabeledsamplesperclass.

•FeatureExtraction:Featureextractionisaprocessofdimensionalityreductionthatinvolvestransformingrawdataintonumericalfeaturesthatcanbeprocessed.

•Featureclustering:Featureclusteringaggregatespointfeaturesintogroupswhosemembersaresimilartoeachotherandnotsimilartomembersofothergroups.

•FeatureRepresentation:RepresentationLearningorfeaturelearningisthesubdisciplineofthe

machinelearningspacethatdealswithextractingfeaturesorunderstandingtherepresentationofadataset.

3

Introduction

•TransferLearning:Pre-trainingamodelonlargeauxiliarydatasetsandthenfine-tuningtheresultingmodelsonthetargettask.Thisisusedforfew-shotlearningsinceonlyafewdatasamplesareavailableinthetarget

domain.

•Transferlearningfromclassicallytrainedmodelsyieldspoorperformanceforfew-shotlearning.Recently,few-shotlearninghasbeenrapidlyimprovedusingmeta-learningmethods.

•Thissuggeststhatthefeaturerepresentationslearnedbymeta-learningmustbefundamentallydifferentfromfeaturerepresentationslearnedthroughconventionaltraining.

•Thispaperunderstandsthedifferencesbetweenfeatureslearnedbymeta-learningandclassicaltraining.

•Basedonthis,thepaperproposessimpleregularizersthatboostfew-shotperformanceappreciably.

4

Meta-LearningFramework

•Inthecontextoffew-shotlearning,theobjectiveofmeta-learningalgorithmsistoproduceanetworkthatquicklyadaptstonewclassesusinglittledata.

•Meta-learningalgorithmsfindparametersthatcanbefine-tunedinafewoptimizationstepsandonafewdatapointsinordertoachievegoodgeneralization.

•Thetaskischaracterizedasn-way,k-shotifthemeta-learningalgorithmmustadapttoclassifydatafromTiafterseeingkexamplesfromeachofthenclassesinTi.

Algorithm

5

6

AlgorithmDescription

•Meta-learningschemestypicallyrelyonbi-leveloptimizationproblemswithaninnerloopandanouterloop.

•Aniterationoftheouterloopinvolvesfirstsamplinga“task,”whichcomprisestwosetsoflabeleddata:thesupportdata,Tis,andthequerydata,Tiq.

•Intheinnerloop,themodelbeingtrainedisfine-tunedusingthesupportdata.

•Fine-tuningproducesnewparametersθi,thatareafunctionoftheoriginalparametersandsupportdata.

•Weevaluatethelossonthequerydataandcomputethegradientsw.r.ttheoriginalparametersθ.Weneedtounrollthefine-tuningstepsandbackpropagatethroughthemtocomputethegradients.

•Finally,theroutinemovesbacktotheouterloop,wherethemeta-learningalgorithmminimizeslossonthequerydatawithrespecttothepre-fine-tunedweights.Basemodelparametersareupdatedusingthe

gradients.

7

Meta-LearningAlgorithms

Avarietyofmeta-learningalgorithmsexist,mostlydifferinginhowtheyarefine-tunedusingthesupportdataduringtheinnerloop:

•MAML:Updatesallnetworkparametersusinggradientdescentduringfine-tuning.

•R2-D2andMetaOptNet:Last-layermeta-learningmethods(onlytrainthelastlayer).Theyfreezethefeatureextractionlayers(featureextractor’sparametersarefrozen)duringtheinnerloop.Onlythelinearclassifierlayeristrainedduringfine-tuning.

•ProtoNet:Last-layermeta-learningmethod.Itclassifiesexamplesbytheproximityoftheirfeaturestothoseofclasscentroids.Theextractedfeaturesareusedtocreateclasscentroidswhichthen

determinethenetwork’sclassboundaries.

8

Few-ShotDatasets

•Mini-ImageNet:ItisaprunedanddownsizedversionoftheImageNetclassificationdataset,

consistingof60,000,84×84RGBcolorimagesfrom100.These100classesaresplitinto64,16,and20classesfortraining,validation,andtestingsets,respectively.

•CIFAR-FSdataset:samplesimagesfromCIFAR-100.CIFAR-FSissplitinthesamewayasmini-ImageNetwith60,00032×32RGBcolorimagesfrom100classesdividedinto64,16,and20

classesfortraining,validation,andtestingsets,respectively.

ComparisonbetweenMeta-LearningandClassicalTrainingModels

•DatasetUsed:1-shotmini-ImageNet

•Classicallytrainedmodelsaretrainedusingcross-entropylossandSGD.

•Commonfine-tuningproceduresareusedforbothmeta-learnedandclassically-trainedmodelsforafaircomparison

•Resultsshowthatmeta-learningmodelsperformbetterthanclassicaltrainingmodelsonfew-shotclassification.

•Thisperformanceadvantageacrosstheboardsuggeststhatmeta-learnedfeaturesarequalitativelydifferentfromconventionalfeaturesandfundamentallysuperiorforfew-shotlearning.

9

10

ClassClusteringinFeatureSpace

MeasuringClusteringinFeatureSpace:

Tomeasurefeatureclustering(FC),weconsidertheintra-classtointer-classvarianceratio:

φi,j-featurevectorcorrespondingtodatapointinclassiintrainingdata

μi-meanoffeaturevectorsinclassi

μ-meanacrossallfeaturevectors

C-numberofclasses

N-numberofdatapointsperclass

Where,fθ(xi,j)=φi,jfθ-featureextractor

xi,j-trainingdatainclassi

Lowvaluesofthisfractioncorrespondtocollectionsoffeaturessuchthatclassesarewell-separatedandahyperplaneformedbychoosingapointfromeachoftwoclassesdoesnotvarydramaticallywiththechoiceofsamples.

WhyClusteringisimportant?

•Asfeaturesinaclassbecomespreadoutandtheclassesarebroughtclosertogether,theclassificationboundariesformedbysamplingone-shotdataoftenmisclassifylargeregions.

•Asfeaturesinaclassarecompactedandclassesmovefarapartfromeachother,theintra-classtointer-classvarianceratiodrops,andthedependenceoftheclassboundaryonthechoiceofone-shotsamplesbecomesweaker.

11

ComparingFeatureRepresentationsofMeta-LearningandClassicallyTrainedModels

•Threeclassesarerandomlychosenfromthetestset,and100samplesaretakenfromeachclass.Thesamplesarethenpassedthroughthefeatureextractor,andtheresultingvectorsareplotted.

•Becausefeaturespaceishigh-dimensional,weperformalinearprojectionontothefirsttwocomponentvectorsdeterminedbyLDA.

•Lineardiscriminantanalysis(LDA)projectsdataontodirectionsthatminimizetheintra-classtointer-classvarianceratio.

•Theclassicallytrainedmodelmashesfeaturestogether,whilethemeta-learnedmodelsdrawtheclassesfartherapart.

12

13

HyperplaneInvariance

Thisregularizerwithonethatpenalizesvariationsinthemaximum-marginhyperplaneseparatingfeaturevectorsin

oppositeclasses

HyperplaneVariationRegularizer:

DatpointsinclassA:x1,x2

DatapointsinclassB:y1,y2

fθ-featureextractor

fθ(x1)-fθ(y1):determinesthedirectionofthemaximum

marginhyperplaneseparatingthetwopointsinthefeaturespace

•Thisfunctionmeasuresthedistancebetweendistancevectorsx1−y1andx2−y2relativetotheirsize.

•Inpractice,duringabatchoftraining,wesamplemanypairsofclassesandtwosamplesfromeachclass.Then,wecomputeRHVonallclasspairsandaddthesetermstothecross-entropyloss.

•WefindthatthisregularizerperformsalmostaswellasFeatureClusteringRegularizerandconclusivelyoutperformsnon-regularizedclassicaltraining.

14

Experiments

•FeatureclusteringandHyperplanevariationvaluesarecomputed.

•Thesetwoquantitiesmeasuretheintra-classtointer-classvarianceratioandinvarianceofseparatinghyperplanes.

•Lowervaluesofeachmeasurementcorrespondtobetterclassseparation.

•OnbothCIFAR-FSandmini-ImageNet,themeta-learnedmodelsattainlowervalues,indicatingthatfeaturespaceclusteringplaysaroleintheeffectivenessofmeta-learning.

15

Experiments

•Weincorporatetheseregularizersintoastandardtrainingroutineoftheclassicaltrainingmodel.

•Inallexperiments,featureclusteringimprovestheperformanceoftransferlearningandsometimesevenachieveshigherperformancethanmeta-learning

16

WeightClustering:FindingClustersofLocalMinimaforTaskLossesinParameterSpace

•SinceReptiledoesnotfixthefeatureextractorduringfine-tuning,itmustfindparametersthatadapteasilytonewtasks.

•WehypothesizethatReptilefindsparametersthatlieveryclosetogoodminimaformanytasksandis,therefore,abletoperformwellonthesetasksafterverylittlefine-tuning.

•ThishypothesisisfurthermotivatedbythecloserelationshipbetweenReptileandconsensusoptimization.

•Inaconsensusmethod,anumberofmodelsareindependentlyoptimizedwiththeirowntask-specificparameters,andthetaskscommunicateviaapenaltythatencouragesalltheindividualsolutionsto

convergearoundacommonvalue.

17

ConsensusFormulation:

•Reptilecanbeinterpretedasapproximatelyminimizingtheconsensusformulation

•Reptiledivergesfromatraditionalconsensusoptimizeronlyinthatitdoesnotexplicitlyconsiderthequadraticpenaltytermwhenminimizingfor˜θp.

18

ConsensusOptimizationImprovesReptile

•WemodifyReptiletoexplicitlyenforceparameterclusteringaroundaconsensusvalue.

•Wefindthatdirectlyoptimizingtheconsensusformulationleadstoimprovedperformance.

•duringeachinnerloopupdatestepinReptile,wepenalizethesquareddistancefromtheparametersforthecurrenttasktotheaverageoftheparametersacrossalltasksinthecurrentbatch.

•ThisisequivalenttotheoriginalReptilewhenα=0.Wecallthismethod“Weight-Clustering.

ReptilewithWeightClusteringRegularizer

n-numberofmeta-trainingsteps

k-numberofiterationsorstepstoperformwithineachmeta-trainingstep

19

20

Resultsofweightclustering

•WecomparetheperformanceofourregularizedReptilealgorithmtothatoftheoriginalReptilemethodaswellasfirst-orderMAML(FOMAML)andaclassicallytrainedmodelofthesamearchitecture.We

testthesemethodsonasampleof100,0005-way1-shotand5-shotmini-ImageNettasks

•ReptilewithWeight-Clusteringachieveshigherperformance.

21

Resultsofweightclustering

•ParametersofnetworkstrainedusingourregularizedversionofReptiledonottravelasfarduringfine-tuningatinferenceasthosetrainedusingvanillaReptile

•Fromthese,weconcludethatourregularizerdoesindeedmovemo