卷积神经网络CNNPPT学习课件

1、CNN的早期历史,卷积神经网络CNNK.Fukushima,“Neocognitron:Aself-organizingneuralnetworkmodelforamechanismofpatternrecognitionunaffectedbyshiftinposition,”BiologicalCybernetics,vol.36,pp.193202,1980Y.LeCun,B.Boser,J.S.Denker,D.Henderson,R.E.Howard,W.Hubbard,andL.D.Jackel,“Backpropagationappliedtohandwrittenzipcode

2、recognition,”NeuralComputation,vol.1,no.4,pp.541551,1989Y.LeCun,L.Bottou,Y.Bengio,andP.Haffner,“Gradient-basedlearningappliedtodocumentrecognition,”ProceedingsoftheIEEE,vol.86,no.11,pp.22782324,1998,1,DL时代的CNN扩展,AKrizhevsky,ISutskever,GEHinton.ImageNetclassificationwithdeepconvolutionalneuralnetwork

3、s.NIPS2012Y.Jiaetal.Caffe:ConvolutionalArchitectureforFastFeatureEmbedding.ACMMM2014K.Simonyan,A.Zisserman.Verydeepconvolutionalnetworksforlarge-scaleimagerecognition.arXivpreprintarXiv:1409.1556,2014C.Szegedy,W.Liu,Y.Jia,P.Sermanet,S.Reed,D.Anguelov,D.Erhan,V.Vanhoucke,A.Rabinovich.Goingdeeperwithc

4、onvolutions.CVPR2015(&arXiv:1409.4842,2014),2,卷积示例,3,卷积形式化,积分形式=常用表达式=()()离散形式一维情况=+()二维情况,=,=,=,=,K称为kernel,4,卷积why?,1.sparseinteractions有限连接，Kernel比输入小连接数少很多，学习难度小，计算复杂度低m个节点与n个节点相连O(mn)限定k(m)个节点与n个节点相连，则为O(kn),5,卷积why?,1.sparseinteractions有限连接，Kernel比输入小连接数少很多，学习难度小，计算复杂度低m个节点与n个节点相连O(mn)限定k(m)个节

5、点与n个节点相连，则为O(kn),6,卷积why?,1.sparseinteractions有限(稀疏)连接Kernel比输入小局部连接连接数少很多学习难度小计算复杂度低层级感受野（生物启发）越高层的神经元，感受野越大,7,卷积why?,2.ParameterSharing（参数共享）Tiedweights进一步极大的缩减参数数量3.Equivariantrepresentations等变性配合Pooling可以获得平移不变性对scale和rotation不具有此属性,8,CNN的基本结构,三个步骤卷积突触前激活，net非线性激活DetectorPoolingLayer的两种定义复杂定义简单定

6、义有些层没有参数,9,Pooling,10,定义（没有需要学习的参数）replacestheoutputofthenetatacertainlocationwithasummarystatisticofthenearbyoutputs种类maxpooling(weighted)averagepooling,WhyPooling?,11,获取不变性小的平移不变性：有即可，不管在哪里很强的先验假设Thefunctionthelayerlearnsmustbeinvarianttosmalltranslations,WhyPooling?,12,获取不变性小的平移不变性：有即可，不管在哪里旋转不变性

7、？9个不同朝向的kernels（模板）,WhyPooling?,13,获取不变性小的平移不变性：有即可，不管在哪里旋转不变性？9个不同朝向的kernels（模板）,Pooling与下采样结合,更好的获取平移不变性更高的计算效率（减少了神经元数）,14,从全连接到有限连接,部分链接权重被强制设置为0通常：非邻接神经元，仅保留相邻的神经元全连接网络的特例，大量连接权重为0,15,WhyConvolution&Pooling？,apriorprobabilitydistributionovertheparametersofamodelthatencodesourbeliefsaboutwhatmod

8、elsarereasonable,beforewehaveseenanydata.模型参数的先验概率分布(Nofreelunch)在见到任何数据之前，我们的信念（经验）告诉我们，什么样的模型参数是合理的Localconnections；对平移的不变性；tiedweigts来自生物神经系统的启发,16,源起：Neocognitron(1980),SimplecomplexLowerorderhighorder,17,K.Fukushima,“Neocognitron:Aself-organizingneuralnetworkmodelforamechanismofpatternrecogniti

9、onunaffectedbyshiftinposition,”BiologicalCybernetics,vol.36,pp.193202,1980,LocalConnection,源起：Neocognitron(1980),18,源起：Neocognitron(1980),训练方法分层自组织competitivelearning无监督输出层独立训练有监督,19,LeCun-CNN1989用于字符识别,简化了Neocognitron的结构训练方法监督训练BP算法正切函数收敛更快，SigmoidLoss，SGD用于邮编识别大量应用,20,LeCun-CNN1989用于字符识别,输入16x16图像

10、L1H112个5x5kernel8x8个神经元L2-H212个5x5x8kernel4x4个神经元L3H330个神经元L4输出层10个神经元总连接数5*5*12*64+5*5*8*12*16+192*30，约66,000个,21,LeCun-CNN1989用于字符识别,Tiedweights对同一个featuremap，kernel对不同位置是相同的！,22,LeCun-CNN1989用于字符识别,23,1998年LeNet数字/字符识别,LeNet-5Featuremapasetofunitswhoseweighsareconstrainedtobeidentical.,24,1998年Le

11、Net数字/字符识别,例如：C3层参数个数(3*6+4*9+6*1)*25+16=1516,25,后续：CNN用于目标检测与识别,26,AlexNetforImageNet(2012),大规模CNN网络650K神经元60M参数使用了各种技巧DropoutDataaugmentReLULocalResponseNormalizationContrastnormalization.,27,Krizhevsky,Alex,IlyaSutskever,andGeoffreyE.Hinton.Imagenetclassificationwithdeepconvolutionalneuralnetwork

12、s.Advancesinneuralinformationprocessingsystems.2012.,AlexNetforImageNet(2012),ReLU激活函数,28,AlexNetforImageNet(2012),实现2块GPU卡输入层150,528其它层253,440186,62464,89664,89643,264409640961000.,29,Krizhevsky,Alex,IlyaSutskever,andGeoffreyE.Hinton.Imagenetclassificationwithdeepconvolutionalneuralnetworks.Advance

13、sinneuralinformationprocessingsystems.2012.,AlexNetforImageNet(2012),ImageNet物体分类任务上1000类，1,431,167幅图像,30,Krizhevsky,Alex,IlyaSutskever,andGeoffreyE.Hinton.Imagenetclassificationwithdeepconvolutionalneuralnetworks.Advancesinneuralinformationprocessingsystems.2012.,AlexNetforImageNet,深度的重要性,31,Krizhe

14、vsky,Alex,IlyaSutskever,andGeoffreyE.Hinton.Imagenetclassificationwithdeepconvolutionalneuralnetworks.Advancesinneuralinformationprocessingsystems.2012.,VGGNet(2014),多个stage每个stage多个卷积层卷积采样间隔1x1卷积核大小3x31个Pooling层(2x2)16-19层多尺度融合,K.Simonyan,A.Zisserman.Verydeepconvolutionalnetworksforlarge-scaleimage

15、recognition.arXivpreprintarXiv:1409.1556,2014,32,VGGNet(2014),几种配置Cov3-64:3x3感受野64个channel,33,K.Simonyan,A.Zisserman.Verydeepconvolutionalnetworksforlarge-scaleimagerecognition.arXivpreprintarXiv:1409.1556,2014,VGGNet(2014),34,K.Simonyan,A.Zisserman.Verydeepconvolutionalnetworksforlarge-scaleimagere

16、cognition.arXivpreprintarXiv:1409.1556,2014,GoogLeNet(2014),超大规模22个卷积层的网络计算复杂度是AlexNet的4倍左右,C.Szegedy,W.Liu,Y.Jia,P.Sermanet,S.Reed,D.Anguelov,D.Erhan,V.Vanhoucke,A.Rabinovich.Goingdeeperwithconvolutions.CVPR2015(&arXiv:1409.4842,2014),35,GoogLeNet(2014),超大规模24层网络Inception结构提取不同scale的特征然后串接起来,C.Szeg

17、edy,W.Liu,Y.Jia,P.Sermanet,S.Reed,D.Anguelov,D.Erhan,V.Vanhoucke,A.Rabinovich.Goingdeeperwithconvolutions.CVPR2015(&arXiv:1409.4842,2014),36,GoogLeNet(2014),超大规模24层网络Inception结构提取不同scale的特征，然后串接起来增加1x1的卷积：把响应图的数量缩小了,C.Szegedy,W.Liu,Y.Jia,P.Sermanet,S.Reed,D.Anguelov,D.Erhan,V.Vanhoucke,A.Rabinovich.Goingdeeperwithconvolutions.CVPR2015(&arXiv:1409.4842,2014),37,GoogLeNet(2014),38,C.Szegedy,W.Liu,Y.Jia,P.Sermanet,S.Reed,D.Anguelov,D.Erhan,V.Vanh