一天搞懂深度学习PPT

DeepLearningTutorial李宏毅Hung-yiLee1

2Deeplearning

attractslotsofattention.Ibelieveyouhaveseenlotsofexcitingresultsbefore.Thistalkfocusesonthebasictechniques.DeeplearningtrendsatGoogle.Source:SIGMOD/JeffDean22023/9/3OutlineLectureI:IntroductionofDeepLearningLectureII:VariantsofNeuralNetworkLectureIII:BeyondSupervisedLearning32023/9/3LectureI:

Introductionof

DeepLearning4

2OutlineIntroductionofDeepLearning“HelloWorld”forDeepLearningTipsforDeepLearning52023/9/3MachineLearning

≈LookingforaFunctionSpeechRecognitionImageRecognitionPlayingGoDialogueSystem“Cat”“Howareyou”“5-5”“Hello”“Hi”(whattheusersaid)(systemresponse)(nextmove)62023/9/3FrameworkAsetoffunction“cat”“dog”“money”“snake”Model“cat”ImageRecognition:72023/9/3FrameworkAsetoffunction“cat”ImageRecognition:ModelTrainingDataGoodnessoffunctionfBetter!“monkey”“cat”“dog”functioninput:functionoutput:SupervisedLearning82023/9/3FrameworkAsetoffunction“cat”ImageRecognition:ModelTrainingDataGoodnessoffunctionf“monkey”“cat”“dog”Pickthe“Best”FunctionUsing“cat”TrainingTestingStep1Step2Step392023/9/3ThreeStepsforDeepLearningStep1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunctionNeuralNetwork102023/9/3NeuralNetwork…biasweightsNeuron………AsimplefunctionActivationfunction112023/9/3NeuralNetworkbiasActivationfunctionweightsNeuron1-2-112-114SigmoidFunction0.98122023/9/3NeuralNetworkDifferentconnectionsleadtodifferentnetworkstructures

Theneuronshavedifferentvaluesofweightsandbiases.132023/9/3FullyConnectFeedforwardNetworkSigmoidFunction1-11-21-1104-20.980.12142023/9/3FullyConnectFeedforwardNetwork1-21-1104-20.980.122-1-1-23-14-10.860.110.620.8300-221-1152023/9/3FullyConnectFeedforwardNetwork1-21-1100.730.52-1-1-23-14-10.720.120.510.8500-22

00Thisisafunction.Inputvector,outputvectorGivennetworkstructure,defineafunctionset162023/9/3OutputLayerHiddenLayersInputLayerFullyConnectFeedforwardNetworkInputOutputLayer1…………Layer2……LayerL…………………………y1y2yMDeepmeansmanyhiddenlayersneuron172023/9/3WhyDeep?UniversalityTheoremReference

forthereason:/chap4.htmlAnycontinuousfunctionfCanberealizedbyanetworkwithonehiddenlayer(givenenoughhiddenneurons)Why“Deep”neuralnetworknot“Fat”neuralnetwork?182023/9/3LogiccircuitsconsistsofgatesAtwolayersoflogicgatescanrepresentanyBooleanfunction.UsingmultiplelayersoflogicgatestobuildsomefunctionsaremuchsimplerNeuralnetworkconsistsofneuronsAhiddenlayernetworkcanrepresentanycontinuousfunction.UsingmultiplelayersofneuronstorepresentsomefunctionsaremuchsimplerlessgatesneededLogiccircuitsNeuralnetworklessparameterslessdata?Morereason:/watch?v=XsC9byQkUH8&list=PLJV_el3uVTsPy9oCRY30oBPNLCo89yu49&index=13WhyDeep?Analogy192023/9/38layers19layers22layersAlexNet(2012)VGG(2014)GoogleNet(2014)16.4%7.3%6.7%/slides/winter1516_lecture8.pdfDeep=Manyhiddenlayers202023/9/3AlexNet(2012)VGG(2014)GoogleNet(2014)152layers3.57%ResidualNet(2015)Taipei101101layers16.4%7.3%6.7%Deep=ManyhiddenlayersSpecialstructure212023/9/3OutputLayerSoftmaxlayerastheoutputlayerOrdinaryLayerIngeneral,theoutputofnetworkcanbeanyvalue.Maynotbeeasytointerpret222023/9/3OutputLayerSoftmaxlayerastheoutputlayerSoftmaxLayer3-312.7200.050.880.12≈0

232023/9/3ExampleApplicationInputOutput16x16=256……Ink→1Noink→0……y1y2y10Eachdimensionrepresentstheconfidenceofadigit.is1is2is0……0.10.70.2Theimageis“2”242023/9/3ExampleApplicationHandwritingDigitRecognitionMachine“2”…………y1y2y10is1is2is0……Whatisneededisafunction……Input:256-dimvectoroutput:10-dimvectorNeuralNetwork252023/9/3OutputLayerHiddenLayersInputLayerExampleApplicationInputOutputLayer1…………Layer2……LayerL……………………“2”……y1y2y10is1is2is0……AfunctionsetcontainingthecandidatesforHandwritingDigitRecognitionYouneedtodecidethenetworkstructuretoletagoodfunctioninyourfunctionset.262023/9/3FAQQ:Howmanylayers?Howmanyneuronsforeachlayer?Q:Canwedesignthenetworkstructure?Q:Canthestructurebeautomaticallydetermined?Yes,butnotwidelystudiedyet.TrialandErrorIntuition+ConvolutionalNeuralNetwork(CNN)inthenextlecture272023/9/3HighwayNetworkResidualNetworkHighwayNetworkDeepResidualLearningforImageRecognition/abs/1512.03385TrainingVeryDeepNetworks/pdf/1507.06228v2.pdf+copycopyGatecontroller282023/9/3InputlayeroutputlayerInputlayeroutputlayerInputlayeroutputlayerHighwayNetworkautomaticallydeterminesthelayersneeded!292023/9/3ThreeStepsforDeepLearningStep1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunction302023/9/3TrainingDataPreparingtrainingdata:imagesandtheirlabelsThelearningtargetisdefinedonthetrainingdata.“5”“0”“4”“1”“3”“1”“2”“9”312023/9/3LearningTarget16x16=256…………………………Ink→1Noink→0……y1y2y10y1hasthemaximumvalueThelearningtargetis……Input:y2hasthemaximumvalueInput:is1is2is0Softmax322023/9/3Loss………………………………y1y2y10

“1”……100……LosscanbesquareerrororcrossentropybetweenthenetworkoutputandtargettargetSoftmaxAscloseaspossibleAgoodfunctionshouldmakethelossofallexamplesassmallaspossible.Givenasetofparameters332023/9/3TotalLossx1x2xRNNNNNN…………y1y2yR

…………x3NNy3

Foralltrainingdata…

TotalLoss:

AssmallaspossibleFindafunctioninfunctionsetthatminimizestotallossL342023/9/3ThreeStepsforDeepLearningStep1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunction352023/9/3Howtopickthebestfunction

EnumerateallpossiblevaluesLayerl……Layerl+1……E.g.speechrecognition:8layersand1000neuronseachlayer1000neurons1000neurons106weightsMillionsofparameters362023/9/3GradientDescent

Random,RBMpre-trainUsuallygoodenough

Pickaninitialvalueforw

372023/9/3GradientDescent

Pickaninitialvalueforw

PositiveNegativeDecreasewIncreasew/album/photo/171572850

382023/9/3GradientDescent

Pickaninitialvalueforw

ηiscalled“learningrate”

Repeat

392023/9/3GradientDescent

Pickaninitialvalueforw

Repeat

(whenupdateislittle)

402023/9/3

GradientDescentColor:ValueofTotalLossLRandomlypickastartingpoint412023/9/3

GradientDescentHopfully,wewouldreachaminima…..

Color:ValueofTotalLossL422023/9/3LocalMinimaTotalLossThevalueofanetworkparameterwVeryslowattheplateauStuckatlocalminima

Stuckatsaddlepoint

432023/9/3LocalMinimaGradientdescentneverguaranteeglobalminima

DifferentinitialpointReachdifferentminima,sodifferentresults442023/9/3GradientDescentThisisthe“learning”ofmachinesindeeplearning……Evenalphagousingthisapproach.Ihopeyouarenottoodisappointed:pPeopleimage……Actually…..452023/9/3Backpropagation

libdnn台大周伯威同學開發Ref:/watch?v=ibJpTrp5mcE462023/9/3Step1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunctionThreeStepsforDeepLearningDeepLearningissosimple……NowIfyouwanttofindafunctionIfyouhavelotsoffunctioninput/output(?)astrainingdataYoucanusedeeplearning472023/9/3Forexample,youcando…….Image

RecognitionNetwork“monkey”“cat”“dog”“monkey”“cat”“dog”482023/9/3Forexample,youcando…….Spamfiltering(/)Network(Yes/No)1/01(Yes)0(No)“free”ine-mail“Talk”ine-mail492023/9/3Forexample,youcando……./Network政治體育經濟“president”indocument“stock”indocument體育政治財經502023/9/3OutlineIntroductionofDeepLearning“HelloWorld”forDeepLearningTipsforDeepLearning512023/9/3Keraskeras.tw/~tlkagk/courses/MLDS_2015_2/Lecture/Theano%20DNN.ecm.mp4/index.html.tw/~tlkagk/courses/MLDS_2015_2/Lecture/RNN%20training%20(v6).ecm.mp4/index.htmlVeryflexibleNeedsomeefforttolearnEasytolearnanduse(stillhavesomeflexibility)YoucanmodifyitifyoucanwriteTensorFloworTheanoInterfaceofTensorFloworTheanoorIfyouwanttolearntheano:522023/9/3KerasFrançoisCholletistheauthorofKeras.HecurrentlyworksforGoogleasadeeplearningengineerandresearcher.Kerasmeans

horn

inGreekDocumentation:http://keras.io/Example:/fchollet/keras/tree/master/examples532023/9/3使用Keras心得感謝沈昇勳同學提供圖檔542023/9/3ExampleApplicationHandwritingDigitRecognitionMachine“1”“Helloworld”fordeeplearningMNISTData:/exdb/mnist/Kerasprovidesdatasetsloadingfunction:http://keras.io/datasets/28x28552023/9/3Kerasy1y2y10……………………Softmax50050028x28562023/9/3Keras572023/9/3KerasStep3.1:ConfigurationStep3.2:Findtheoptimalnetworkparameters

0.1Trainingdata(Images)Labels(digits)582023/9/3KerasStep3.2:Findtheoptimalnetworkparameters/versions/r0.8/tutorials/mnist/beginners/index.htmlNumberoftrainingexamplesnumpyarray28x28=784numpyarray10Numberoftrainingexamples…………592023/9/3Kerashttp://keras.io/getting-started/faq/#how-can-i-save-a-keras-modelHowtousetheneuralnetwork(testing):case1:case2:Saveandloadmodels602023/9/3KerasUsingGPUtospeedtrainingWay1THEANO_FLAGS=device=gpu0pythonYourCode.pyWay2(inyourcode)importosos.environ["THEANO_FLAGS"]="device=gpu0"612023/9/3Demo622023/9/3Step1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunctionThreeStepsforDeepLearningDeepLearningissosimple……632023/9/3OutlineIntroductionofDeepLearning“HelloWorld”forDeepLearningTipsforDeepLearning642023/9/3NeuralNetworkGoodResultsonTestingData?GoodResultsonTrainingData?Step1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunctionYESYESNONOOverfitting!RecipeofDeepLearning652023/9/3DonotalwaysblameOverfittingDeepResidualLearningforImageRecognition/abs/1512.03385TestingDataOverfitting?TrainingDataNotwelltrained662023/9/3NeuralNetworkGoodResultsonTestingData?GoodResultsonTrainingData?YESYESRecipeofDeepLearningDifferentapproachesfordifferentproblems.e.g.dropoutforgoodresultsontestingdata672023/9/3GoodResultsonTestingData?GoodResultsonTrainingData?YESYESRecipeofDeepLearningChoosingproperlossMini-batchNewactivationfunctionAdaptiveLearningRateMomentum682023/9/3ChoosingProperLoss………………………………y1y2y10loss“1”……100……targetSoftmax

SquareErrorCrossEntropy

Whichoneisbetter?

……100=0=0692023/9/3DemoSquareErrorCrossEntropySeveralalternatives:https://keras.io/objectives/702023/9/3Demo712023/9/3ChoosingProperLossTotalLossw1w2CrossEntropySquareErrorWhenusingsoftmaxoutputlayer,choosecrossentropy/proceedings/papers/v9/glorot10a/glorot10a.pdf722023/9/3GoodResultsonTestingData?GoodResultsonTrainingData?YESYESRecipeofDeepLearningChoosingproperlossMini-batchNewactivationfunctionAdaptiveLearningRateMomentum732023/9/3Mini-batchx1NN……y1

x31NNy31

x2NN……y2

x16NNy16

Pickthe1stbatchRandomlyinitializenetworkparametersPickthe2ndbatchMini-batchMini-batch

UpdateparametersonceUpdateparametersonceUntilallmini-batcheshavebeenpicked…oneepochRepeattheaboveprocessWedonotreallyminimizetotalloss!742023/9/3Mini-batchx1NN……y1

x31NNy31

Mini-batchPickthe1stbatchPickthe2ndbatch

UpdateparametersonceUpdateparametersonceUntilallmini-batcheshavebeenpicked…oneepoch100examplesinamini-batchRepeat20times752023/9/3Mini-batchOriginalGradientDescentWithMini-batchUnstable!!!Thecolorsrepresentthetotalloss.762023/9/3Mini-batchisFaster1epochSeeallexamplesSeeonlyonebatchUpdateafterseeingallexamplesIfthereare20batches,update20timesinoneepoch.OriginalGradientDescentWithMini-batchNotalwaystruewithparallelcomputing.Canhavethesamespeed(notsuperlargedataset)Mini-batchhasbetterperformance!772023/9/3Demo782023/9/3x1NN……y1

x31NNy31

x2NN……y2

x16NNy16

Mini-batchMini-batchShufflethetrainingexamplesforeachepochEpoch1x1NN……y1

x17NNy17

x2NN……y2

x26NNy26

Mini-batchMini-batchEpoch2Don’tworry.ThisisthedefaultofKeras.792023/9/3GoodResultsonTestingData?GoodResultsonTrainingData?YESYESRecipeofDeepLearningChoosingproperlossMini-batchNewactivationfunctionAdaptiveLearningRateMomentum802023/9/3HardtogetthepowerofDeep…Deeperusuallydoesnotimplybetter.ResultsonTrainingData812023/9/3Demo822023/9/3VanishingGradientProblemLargergradientsAlmostrandomAlreadyconvergebasedonrandom!?LearnveryslowLearnveryfast…………………………………………y1y2yMSmallergradients832023/9/3VanishingGradientProblem…………………………………………

……

Intuitivewaytocomputethederivatives…

SmallergradientsLargeinputSmalloutput842023/9/3HardtogetthepowerofDeep…In2006,peopleusedRBMpre-training.In2015,peopleuseReLU.852023/9/3ReLURectifiedLinearUnit(ReLU)Reason:1.Fasttocompute2.Biologicalreason3.Infinitesigmoidwithdifferentbiases4.Vanishinggradientproblem

[XavierGlorot,AISTATS’11][AndrewL.Maas,ICML’13][KaimingHe,arXiv’15]862023/9/3ReLU0000

872023/9/3ReLUAThinnerlinearnetworkDonothavesmallergradients

882023/9/3Demo892023/9/3ReLU-variant

αalsolearnedbygradientdescent902023/9/3MaxoutLearnableactivationfunction[IanJ.Goodfellow,ICML’13]MaxInputMax+

+

+

+

MaxMax+

+

+

+

ReLUisaspecialcasesofMaxoutYoucanhavemorethan2elementsinagroup.neuron912023/9/3MaxoutLearnableactivationfunction[IanJ.Goodfellow,ICML’13]ActivationfunctioninmaxoutnetworkcanbeanypiecewiselinearconvexfunctionHowmanypiecesdependingonhowmanyelementsinagroupReLUisaspecialcasesofMaxout2elementsinagroup3elementsinagroup922023/9/3GoodResultsonTestingData?GoodResultsonTrainingData?YESYESRecipeofDeepLearningChoosingproperlossMini-batchNewactivationfunctionAdaptiveLearningRateMomentum932023/9/3

LearningRatesIflearningrateistoolargeTotallossmaynotdecreaseaftereachupdateSetthelearningrateηcarefully942023/9/3

LearningRatesIflearningrateistoolargeSetthelearningrateηcarefullyIflearningrateistoosmallTrainingwouldbetooslowTotallossmaynotdecreaseaftereachupdate952023/9/3LearningRates

962023/9/3AdagradParameterdependentlearningrate

constant

Summationofthesquareofthepreviousderivatives

Original:Adagrad:972023/9/3Adagradg0g1……0.10.2……g0g1……20.010.0……Observation:1.Learningrateissmallerandsmallerforallparameters2.Smallerderivatives,largerlearningrate,andviceversa

Why?

Learningrate:Learningrate:

982023/9/3SmallerDerivativesLargerLearningRate2.Smallerderivatives,largerlearningrate,andviceversaWhy?SmallerLearningRateLargerderivatives992023/9/3Notthewholestory……Adagrad[JohnDuchi,JMLR’11]RMSprop/watch?v=O3sxAc4hxZUAdadelta[MatthewD.Zeiler,arXiv’12]“Nomorepeskylearningrates”[TomSchaul,arXiv’12]AdaSecant[CaglarGulcehre,arXiv’14]Adam

[DiederikP.Kingma,ICLR’15]Nadam

/proj2015/054_report.pdf

1002023/9/3GoodResultsonTestingData?GoodResultsonTrainingData?YESYESRecipeofDeepLearningChoosingproperlossMini-batchNewactivationfunctionAdaptiveLearningRateMomentum1012023/9/3Hardtofind

optimalnetworkparametersTotalLossThevalueofanetworkparameterwVeryslowattheplateauStuckatlocalminima

Stuckatsaddlepoint

1022023/9/3Inphysicalworld

……MomentumHowaboutputthisphenomenoningradientdescent?1032023/9/3Movement=Negativeof𝜕𝐿∕𝜕𝑤+MomentumMomentumcost𝜕𝐿∕𝜕𝑤=0Stillnotguaranteereachingglobalminima,butgivesomehope……

MomentumRealMovement1042023/9/3AdamRMSProp(AdvancedAdagrad)+Momentum1052023/9/3Demo1062023/9/3GoodResultsonTestingData?GoodResultsonTrainingData?YESYESRecipeofDeepLearningEarlyStoppingRegularizationDropoutNetworkStructure1072023/9/3PanaceaforOverfittingHavemoretrainingdataCreatemoretrainingdata(?)OriginalTrainingData:CreatedTrainingData:Shift15。Handwritingrecognition:1082023/9/3GoodResultsonTestingData?GoodResultsonTrainingData?YESYESRecipeofDeepLearningEarlyStoppingRegularizationDropoutNetworkStructure1092023/9/3DropoutTraining:EachtimebeforeupdatingtheparametersEachneuronhasp%todropout1102023/9/3DropoutTraining:EachtimebeforeupdatingtheparametersEachneuronhasp%todropoutUsingthenewnetworkfortrainingThestructureofthenetworkischanged.Thinner!Foreachmini-batch,weresamplethedropoutneurons1112023/9/3DropoutTesting:NodropoutIfthedropoutrateattrainingisp%,alltheweightstimes1-p%

1122023/9/3Dropout-IntuitiveReasonTrainingTestingDropout(腳上綁重物)Nodropout(拿下重物後就變很強)1132023/9/3Dropout-IntuitiveReasonWhytheweightsshouldmultiply(1-p)%(dropoutrate)whentesting?TrainingofDropoutTestingofDropout

Assumedropoutrateis50%

NodropoutWeightsfromtraining

Weightsmultiply1-p%1142023/9/3Dropoutisakindofensemble.EnsembleNetwork1Network2Network3Network4TrainabunchofnetworkswithdifferentstructuresTrainingSetSet

1Set2Set3Set41152023/9/3Dropoutisakindofensemble.Ensembley1Network1Network2Network3Network4Testingdataxy2y3y4average1162023/9/3Dropoutisakindofensemble.TrainingofDropoutminibatch1……Usingonemini-batchtotrainonenetworkSomeparametersinthenetworkaresharedminibatch2minibatch3minibatch4Mneurons2Mpossiblenetworks1172023/9/3Dropoutisakindofensemble.testingdataxTestingofDropout……averagey1y2y3Alltheweightsmultiply1-p%≈y?????1182023/9/3MoreaboutdropoutMorereferencefordropout[NitishSrivastava,JMLR’14][PierreBaldi,NIPS’13][GeoffreyE.Hinton,arXiv’12]DropoutworksbetterwithMaxout[IanJ.Goodfellow,ICML’13]Dropconnect[LiWan,ICML’13]DropoutdeleteneuronsDropconnectdeletestheconnectionbetweenneuronsAnnealeddropout[S.J.Rennie,SLT’14]DropoutratedecreasesbyepochsStandout[J.Ba,NISP’13]Eachneuralhasdifferentdropoutrate1192023/9/3Demoy1y2y10……………………Softmax500500model.add(dropout(0.8))model.add(dropout(0.8))1202023/9/3Demo1212023/9/3GoodResultsonTestingData?GoodResultsonTrainingData?YESYESRecipeofDeepLearningEarlyStoppingRegularizationDropoutNetworkStructureCNNisaverygoodexample!(nextlecture)1222023/9/3ConcludingRemarks123

2RecipeofDeepLearningNeuralNetworkGoodResultsonTestingData?GoodResultsonTrainingData?Step1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunctionYESYESNONO1242023/9/3LectureII:

VariantsofNeuralNetworks125

2VariantsofNeuralNetworksConvolutionalNeuralNetwork(CNN)RecurrentNeuralNetwork(RNN)Widelyusedinimageprocessing1262023/9/3WhyCNNforImage?Canthenetworkbesimplifiedbyconsideringthepropertiesofimages?……………………………………ThemostbasicclassifiersUse1stlayerasmoduletobuildclassifiersUse2ndlayerasmodule……[Zeiler,M.D.,ECCV2014]Representedaspixels1272023/9/3WhyCNNforImageSomepatternsaremuchsmallerthanthewholeimageAneurondoesnothavetoseethewholeimagetodiscoverthepattern.“beak”

detectorConnectingtosmallregionwithlessparameters1282023/9/3WhyCNNforImageThesamepatternsappearindifferentregions.“upper-leftbeak”

detector“middlebeak”

detectorTheycanusethesamesetofparameters.Doalmostthesamething1292023/9/3WhyCNNforImageSubsampling

thepixelswillnotchangetheobjectsubsamplingbirdbirdWecansubsamplethepixelstomakeimagesmallerLessparametersforthenetworktoprocesstheimage1302023/9/3Step1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunctionThreeStepsforDeepLearningDeepLearningissosimple……ConvolutionalNeuralNetwork1312023/9/3ThewholeCNNFullyConnectedFeedforwardnetworkcatdog……ConvolutionMaxPoolingConvolutionMaxPoolingFlattenCanrepeatmanytimes1322023/9/3ThewholeCNNConvolutionMaxPoolingConvolutionMaxPoolingFlattenCanrepeatmanytimesSomepatternsaremuchsmallerthanthewholeimageThesamepatternsappearindifferentregions.Subsampling

thepixelswillnotchangetheobjectProperty1Property2Property31332023/9/3ThewholeCNNFullyConnectedFeedforwardnetworkcatdog……ConvolutionMaxPoolingConvolutionMaxPoolingFlattenCanrepeatmanytimes1342023/9/3CNN–Convolution1000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter1-11-1-11-1-11-1Filter2……Thosearethenetworkparameterstobelearned.MatrixMatrixEachfilterdetectsasmallpattern(3x3).Property11352023/9/3CNN–Convolution1000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter13-1stride=11362023/9/3CNN–Convolution1000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter13-3Ifstride=2Wesetstride=1below1372023/9/3CNN–Convolution1000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter13-1-3-1-310-3-3-3013-2-2-1stride=1Property21382023/9/3CNN–Convolution1000010100100011001000100100100010106x6image3-1-3-1-310-3-3-3013-2-2-1-11-1-11-1-11-1Filter2-1-1-1-1-1-1-21-1-1-21-10-43Dothesameprocessforeveryfilterstride=14x4imageFeatureMap1392023/9/3CNN–ZeroPadding1000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter1Youwillgetanother6x6imagesinthisway0Zeropadding0000000001402023/9/3CNN–Colorfulimage1000010100100011001000100100100010101000010100100011001000100100100010101000010100100011001000100100100010101-1-1-11-1-1-11Filter1-11-1-11-1-11-1Filter21-1-1-11-1-1-111-1-1-11-1-1-11-11-1-11-1-11-1-11-1-11-1-11-1Colorfulimage1412023/9/3100001010010001100100010010010001010imageconvolution-11-1-11-1-11-11-1-1-11-1-1-11…………100001010010001100100010010010001010Convolutionv.s.FullyConnectedFully-connected1422023/9/31000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter11:2:3:…7:8:9:…13:14:15:…Onlyconnectto9input,notfullyconnected4:10:16:1000010000113Lessparameters!1432023/9/31000010100100011001000100100100010101-1-1-11-1-1-11Filter11:2:3:…7:8:9:…13:14:15:…4:10:16:1000010000113-1Sharedweights6x6imageLessparameters!Evenlessparameters!1442023/9/3ThewholeCNNFullyConnectedFeedforwardnetworkcatdog……ConvolutionMaxPoolingConvolutionMaxPoolingFlattenCanrepeatmanytimes1452023/9/3CNN–MaxPooling3-1-3-1-310-3-3-3013-2-2-1-11-1-11-1-11-1Filter2-1-1-1-1-1-1-21-1-1-21-10-431-1-1-11-1-1-11Filter11462023/9/3CNN–MaxPooling1000010100100011001000100100100010106x6image3013-11302x2imageEachfilterisachannelNewimagebutsmallerConvMaxPooling1472023/9/3ThewholeCNNConvolutionMaxPoolingConvolutionMaxPoolingCanrepeatmanytimesAnewimageThenumberofthechannelisthenumberoffiltersSmallerthantheoriginalimage3013-11301482023/9/3ThewholeCNNFullyConnectedFeedforwardnetworkcatdog……ConvolutionMaxPoolingConvolutionMaxPoolingFlattenAnewimageAnewimage1492023/9/3Flatten3013-1130Flatten3013-1103FullyConnectedFeedforwardnetwork1502023/9/3ConvolutionalNeuralNetworkLearning:Nothingspecial,justgradientdescent……CNN“monkey”“cat”“dog”Convolution,MaxPooling,fullyconnected100……targetStep1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunctionConvolutionalNeuralNetwork1512023/9/3Onlymodifiedthenetworkstructureandinputformat(vector->3-Dtensor)CNNinKerasConvolutionMaxPoolingConvolutionMaxPoolinginput1-1-1-11-1-1-11-11-1-11-1-11-1Thereare25

3x3filters.……Input_shape=(1,28,28)1:black/weight,3:RGB28x28pixels3-1-3131522023/9/3Onlymodifiedthenetworkstructureandinputformat(vector->3-Dtensor)CNNinKerasConvolutionMaxPoolingConvolutionMaxPoolinginput1x28x2825x26x2625x13x1350x11x1150x5x5Howmanyparametersforeachfilter?Howmanyparametersforeachfilter?92251532023/9/3Onlymodifiedthenetworkstructureandinputformat(vector->3-Dtensor)CNNinKerasConvolutionMaxPoolingConvolutionMaxPoolinginput1x28x2825