neuralnets-2012-001 06_Lecture6 01_Overview_of_mini-batch_gradient_descent

NeuralNetworksforMachineLearningLecture6aOverviewofmini batchgradientdescent GeoffreyHintonwithNitishSrivastavaKevinSwersky Reminder Theerrorsurfaceforalinearneuron Theerrorsurfaceliesinaspacewithahorizontalaxisforeachweightandoneverticalaxisfortheerror Foralinearneuronwithasquarederror itisaquadraticbowl Verticalcross sectionsareparabolas Horizontalcross sectionsareellipses Formulti layer non linearnetstheerrorsurfaceismuchmorecomplicated Butlocally apieceofaquadraticbowlisusuallyaverygoodapproximation E w1 w2 Convergencespeedoffullbatchlearningwhentheerrorsurfaceisaquadraticbowl Goingdownhillreducestheerror butthedirectionofsteepestdescentdoesnotpointattheminimumunlesstheellipseisacircle Thegradientisbiginthedirectioninwhichweonlywanttotravelasmalldistance Thegradientissmallinthedirectioninwhichwewanttotravelalargedistance Evenfornon linearmulti layernets theerrorsurfaceislocallyquadratic sothesamespeedissuesapply Howthelearninggoeswrong Ifthelearningrateisbig theweightssloshtoandfroacrosstheravine Ifthelearningrateistoobig thisoscillationdiverges Whatwewouldliketoachieve Movequicklyindirectionswithsmallbutconsistentgradients Moveslowlyindirectionswithbigbutinconsistentgradients E w Stochasticgradientdescent Ifthedatasetishighlyredundant thegradientonthefirsthalfisalmostidenticaltothegradientonthesecondhalf Soinsteadofcomputingthefullgradient updatetheweightsusingthegradientonthefirsthalfandthengetagradientforthenewweightsonthesecondhalf Theextremeversionofthisapproachupdatesweightsaftereachcase Itscalled online Mini batchesareusuallybetterthanonline Lesscomputationisusedupdatingtheweights Computingthegradientformanycasessimultaneouslyusesmatrix matrixmultiplieswhichareveryefficient especiallyonGPUsMini batchesneedtobebalancedforclasses Twotypesoflearningalgorithm Ifweusethefullgradientcomputedfromallthetrainingcases therearemanycleverwaystospeeduplearning e g non linearconjugategradient Theoptimizationcommunityhasstudiedthegeneralproblemofoptimizingsmoothnon linearfunctionsformanyyears Multilayerneuralnetsarenottypicaloftheproblemstheystudysotheirmethodsmayneedalotofadaptation Forlargeneuralnetworkswithverylargeandhighlyredundanttrainingsets itisnearlyalwaysbesttousemini batchlearning Themini batchesmayneedtobequitebigwhenadaptingfancymethods Bigmini batchesaremorecomputationallyefficient Abasicmini batchgradientdescentalgorithm Guessaninitiallearningrate Iftheerrorkeepsgettingworseoroscillateswildly reducethelearningrate Iftheerrorisfallingfairlyconsistentlybutslowly increasethelearningrate Writeasimpleprogramtoautomatethiswayofadjustingthelearningrate Towardstheendofmini batchlearningitnearlyalwayshelpstoturndownthelearningrate Thisremovesfluctuationsinthefinalweightscausedbythevariationsbetweenmini batches Turndownthelearningratewhentheerrorstopsdecreasing Usetheerroronaseparatevalidationset NeuralNetworksforMachineLearningLecture6bAbagoftricksformini batchgradientdescent GeoffreyHintonwithNitishSrivastavaKevinSwersky Becarefulaboutturningdownthelearningrate Turningdownthelearningratereducestherandomfluctuationsintheerrorduetothedifferentgradientsondifferentmini batches Sowegetaquickwin Butthenwegetslowerlearning Don tturndownthelearningratetoosoon error epoch reducelearningrate Initializingtheweights Iftwohiddenunitshaveexactlythesamebiasandexactlythesameincomingandoutgoingweights theywillalwaysgetexactlythesamegradient Sotheycanneverlearntobedifferentfeatures Webreaksymmetrybyinitializingtheweightstohavesmallrandomvalues Ifahiddenunithasabigfan in smallchangesonmanyofitsincomingweightscancausethelearningtoovershoot Wegenerallywantsmallerincomingweightswhenthefan inisbig soinitializetheweightstobeproportionaltosqrt fan in Wecanalsoscalethelearningratethesameway Shiftingtheinputs Whenusingsteepestdescent shiftingtheinputvaluesmakesabigdifference Itusuallyhelpstotransformeachcomponentoftheinputvectorsothatithaszeromeanoverthewholetrainingset Thehypberbolictangent whichis2 logistic 1 produceshiddenactivationsthatareroughlyzeromean Inthisrespectitsbetterthanthelogistic 101 101 2101 99 0 giveserrorsurface 1 1 21 1 0 giveserrorsurface colorindicatestrainingcase Scalingtheinputs Whenusingsteepestdescent scalingtheinputvaluesmakesabigdifference Itusuallyhelpstotransformeachcomponentoftheinputvectorsothatithasunitvarianceoverthewholetrainingset 1 1 21 1 0 0 1 10 20 1 10 0 giveserrorsurface giveserrorsurface colorindicatesweightaxis Amorethoroughmethod Decorrelatetheinputcomponents Foralinearneuron wegetabigwinbydecorrelatingeachcomponentoftheinputfromtheotherinputcomponents Thereareseveraldifferentwaystodecorrelateinputs AreasonablemethodistousePrincipalComponentsAnalysis Droptheprincipalcomponentswiththesmallesteigenvalues Thisachievessomedimensionalityreduction Dividetheremainingprincipalcomponentsbythesquarerootsoftheireigenvalues Foralinearneuron thisconvertsanaxisalignedellipticalerrorsurfaceintoacircularone Foracircularerrorsurface thegradientpointsstraighttowardstheminimum Commonproblemsthatoccurinmultilayernetworks Ifwestartwithaverybiglearningrate theweightsofeachhiddenunitwillallbecomeverybigandpositiveorverybigandnegative Theerrorderivativesforthehiddenunitswillallbecometinyandtheerrorwillnotdecrease Thisisusuallyaplateau butpeopleoftenmistakeitforalocalminimum Inclassificationnetworksthatuseasquarederrororacross entropyerror thebestguessingstrategyistomakeeachoutputunitalwaysproduceanoutputequaltotheproportionoftimeitshouldbea1 Thenetworkfindsthisstrategyquicklyandmaytakealongtimetoimproveonitbymakinguseoftheinput Thisisanotherplateauthatlookslikealocalminimum Fourwaystospeedupmini batchlearning Use momentum Insteadofusingthegradienttochangethepositionoftheweight particle useittochangethevelocity UseseparateadaptivelearningratesforeachparameterSlowlyadjusttherateusingtheconsistencyofthegradientforthatparameter rmsprop Dividethelearningrateforaweightbyarunningaverageofthemagnitudesofrecentgradientsforthatweight Thisisthemini batchversionofjustusingthesignofthegradient Takeafancymethodfromtheoptimizationliteraturethatmakesuseofcurvatureinformation notthislecture AdaptittoworkforneuralnetsAdaptittoworkformini batches NeuralNetworksforMachineLearningLecture6cThemomentummethod GeoffreyHintonwithNitishSrivastavaKevinSwersky Theintuitionbehindthemomentummethod Imagineaballontheerrorsurface Thelocationoftheballinthehorizontalplanerepresentstheweightvector Theballstartsoffbyfollowingthegradient butonceithasvelocity itnolongerdoessteepestdescent Itsmomentummakesitkeepgoinginthepreviousdirection Itdampsoscillationsindirectionsofhighcurvaturebycombininggradientswithoppositesigns Itbuildsupspeedindirectionswithagentlebutconsistentgradient Theequationsofthemomentummethod Theeffectofthegradientistoincrementthepreviousvelocity Thevelocityalsodecaysbyawhichisslightlylessthen1 Theweightchangeisequaltothecurrentvelocity Theweightchangecanbeexpressedintermsofthepreviousweightchangeandthecurrentgradient Thebehaviorofthemomentummethod Iftheerrorsurfaceisatiltedplane theballreachesaterminalvelocity Ifthemomentumiscloseto1 thisismuchfasterthansimplegradientdescent Atthebeginningoflearningtheremaybeverylargegradients Soitpaystouseasmallmomentum e g 0 5 Oncethelargegradientshavedisappearedandtheweightsarestuckinaravinethemomentumcanbesmoothlyraisedtoitsfinalvalue e g 0 9oreven0 99 Thisallowsustolearnataratethatwouldcausedivergentoscillationswithoutthemomentum Abettertypeofmomentum Nesterov1983 Thestandardmomentummethodfirstcomputesthegradientatthecurrentlocationandthentakesabigjumpinthedirectionoftheupdatedaccumulatedgradient IlyaSutskever 2012unpublished suggestedanewformofmomentumthatoftenworksbetter InspiredbytheNesterovmethodforoptimizingconvexfunctions Firstmakeabigjumpinthedirectionofthepreviousaccumulatedgradient Thenmeasurethegradientwhereyouendupandmakeacorrection Itsbettertocorrectamistakeafteryouhavemadeit ApictureoftheNesterovmethod Firstmakeabigjumpinthedirectionofthepreviousaccumulatedgradient Thenmeasurethegradientwhereyouendupandmakeacorrection brownvector jump redvector correction greenvector accumulatedgradientbluevectors standardmomentum NeuralNetworksforMachineLearningLecture6dAseparate adaptivelearningrateforeachconnection GeoffreyHintonwithNitishSrivastavaKevinSwersky Theintuitionbehindseparateadaptivelearningrates Inamultilayernet theappropriatelearningratescanvarywidelybetweenweights Themagnitudesofthegradientsareoftenverydifferentfordifferentlayers especiallyiftheinitialweightsaresmall Thefan inofaunitdeterminesthesizeofthe overshoot effectscausedbysimultaneouslychangingmanyoftheincomingweightsofaunittocorrectthesameerror Souseagloballearningrate setbyhand multipliedbyanappropriatelocalgainthatisdeterminedempiricallyforeachweight Gradientscangetverysmallintheearlylayersofverydeepnets Thefan inoftenvarieswidelybetweenlayers Onewaytodeterminetheindividuallearningrates Startwithalocalgainof1foreveryweight Increasethelocalgainifthegradientforthatweightdoesnotchangesign Usesmalladditiveincreasesandmultiplicativedecreases formini batch Thisensuresthatbiggainsdecayrapidlywhenoscillationsstart Ifthegradientistotallyrandomthegainwillhoveraround1whenweincreasebyplushalfthetimeanddecreasebytimeshalfthetime Tricksformakingadaptivelearningratesworkbetter Limitthegainstolieinsomereasonablerangee g 0 1 10 or 01 100 Usefullbatchlearningorbigmini batchesThisensuresthatchangesinthesignofthegradientarenotmainlyduetothesamplingerrorofamini batch Adaptivelearningratescanbecombinedwithmomentum Usetheagreementinsignbetweenthecurrentgradientforaweightandthevelocityforthatweight Jacobs 1989 Adaptivelearningratesonlydealwithaxis alignedeffects Momentumdoesnotcareaboutthealignmentoftheaxes NeuralNetworksforMachineLearningLecture6ermsprop Dividethegradientbyarunningaverageofitsrecentmagnitude GeoffreyHintonwithNitishSrivastavaKevinSwersky rprop Usingonlythesignofthegradient Themagnitudeofthegradientcanbeverydifferentfordifferentweightsandcanchangeduringlearning Thismakesithardtochooseasinglegloballearningrate Forfullbatchlearning wecandealwiththisvariationbyonlyusingthesignofthegradient Theweightupdatesareallofthesamemagnitude Thisescapesfromplateauswithtinygradientsquickly rprop Thiscombinestheideaofonlyusingthesignofthegradientwiththeideaofadaptingthestepsizeseparatelyforeachweight Increasethestepsizeforaweightmultiplicatively e g times1 2 ifthesignsofitslasttwogradientsagree Otherwisedecreasethestepsizemultiplicatively e g times0 5 Limitthestepsizestobelessthan50andmorethanamillionth MikeShuster sadvice Whyrpropdoesnotworkwithmini batches Theideabehindstochasticgradientdescentisthatwhenthelearningrateissmall itaveragesthegradientsoversuccessivemini batches Consideraweightthatgetsagradientof 0 1onninemini batchesandagradientof 0 9onthetenthmini batch Wewantthisweighttostayroughlywhereitis rpropwouldincrementtheweightninetimesanddecrementitoncebyaboutthesameamount assuminganyadaptationofthestepsizesissmallonthistime scale Sotheweightwouldgrowalot Isthereawaytocombine Therobustnessofrprop Theefficiencyofmini batches Theeffectiveaveragingofgradientsovermini batches rmsprop Amini batchversionofrprop rpropisequivalenttousingthegradientbutalsodividingbythesizeofthegradient Theproblemwithmini batchrpropisthatwed