CNN卷积神经网络原理

CNN卷积神经网络原理CNN的最大特点就是稀疏连接(局部感受)和权值共享，如下面两图所示，左为稀疏连接，右为权值共享。稀疏连接和权值共享可以减少 (convolution运算)，C1到S2是一个子采样层(pooling运算)，关于卷积和子采样的具体过程可以参考下图：全连接的，这就相当于一个MLP的隐含层了(如果你不清楚MLP，多少卷积层和子采样层、采用什么分类器。当了结构以后，如何求解层与层之间的连接参数一般采用向前传播(FP)+向后传播(BP)的方法来训练。具体可参考上面给出的链二、CNN卷积神经网络代码详细解读(基于python+theano)教程：，这个代码实现的是一个简化了的••••locationspecificgainandbiasparameters另外，代码里将卷积层和子采用层合在一起，定义为“LeNetConvPoolLayer“(卷积采样层)，这好理解，因为它们总个地方需要注意，代码中将卷积后的输出直接作 (1)导入必要的模块[python]1.importcPickle2.importgzip3.importos4.importsys5.importtime6.importnumpy7.importtheano8.importasT9.fromimportdownsample10.fromimportconv (2)定义CNN的基本"构件"CNN的基本构件包括卷积采样层、隐含层、分类器，如下见代码注释：[python]2.卷积+下采样合成一个层LeNetConvPoolLayer3.rng:随机数生成器，用于初始化W4.filter_shape:(numberoffilters,numinputfeaturemaps,filterheight,filterwidth)5.image_shape:(batchsize,numinputfeaturemaps,imageheight,imagewidth)6.poolsize:(#rows,#cols)8.classLeNetConvPoolLayer(object):9.def__init__(self,rng,input,filter_shape,image_shape,poolsize=(2,2)):11.#image_shape[1]和filter_shape[1]都是numinputfeaturemaps，它们必须是一样的。12.assertimage_shape[1]==filter_shape[1]t14.#每个隐层神经元(即像素)与上一层的连接数为numinputfeaturemaps*filterheight*filterwidth。15.#可以用(filter_shape[1:])来求得16.fan_in=(filter_shape[1:])17.#lowerlayer上每个神经元获得的梯度来自于："numoutputfeaturemaps*filterheight*filterwidth"/poolingsize18.fan_out=(filter_shape[0]*(filter_shape[2:])/19.(poolsize))20.#以上求得fan_in、fan_out，将它们代入公式，以此来随机初始化W,W就是线性卷积核21.W_bound=(6./(fan_in+fan_out))3.(24.(low=-W_bound,high=W_bound,size=filter_shape),26.borrow=True28.#thebiasisa1Dtensor--onebiasperoutputfeaturemap29.#偏置b是一维向量，每个输出图的特征图都对应一个偏置，30.#而输出的特征图的个数由filter个数决定，因此用filter_shape[0]即numberoffilters来初始化31.b_values=((filter_shape[032.=(value=b_values,borrow=True)33.#将输入图像与filter卷积，函数35.conv_out=(36.input=input,37.filters=,38.filter_shape=filter_shape,39.image_shape=image_shape40.)42.pooled_out=(43.input=conv_out,44.ds=poolsize,45.ignore_border=True46.)47.#加偏置，再通过tanh映射，得到卷积+子采样层的最终输出b一维向量，这里用维度转换函数dimshuffle将其reshape。比如b是(10,)，49.#则('x',0,'x','x'))将其reshape为(1,10,1,1)51.#卷积+采样层的参数篇文章《》中的HiddenLayer是一致的，直接拿过来：[python]2.注释：3.这是定义隐藏层的类，首先明确：隐藏层的输入即input，输出即隐藏层的神经元个数。输入层与隐藏层是全连接的。4.假设输入是n_in维的向量(也可以说时n_in个神经元)，隐藏层有n_out个神经元，则因为是全连接，5.一共有n_in*n_out个权重，故W大小时(n_in,n_out),n_in行n_out列，每一列对应隐藏层的每一个神经元的连接权重。bnout故b时n_out维向量。7.input训练模型所用到的所有输入，并不是MLP的输入层，MLP的输入层的神经元个数时n_in，而这里的参数input大小是(n_example,n_in),每一行一个样本，即每一行作为MLP8.activation:激活函数,这里定义为函数tanh10.classHiddenLayer(object):11.def__init__(self,rng,input,n_in,n_out,W=None,b=None,12.activation=:13.=input#类HiddenLayer的input即所传递进来的input15.注释：16.另外，W的初始化有个规则：如果使用tanh函数，则在-sqrt(6./(n_in+n_hidden))到sqrt(6./(n_in+n_hidden))之间均匀#如果W未初始化，则根据上述方法初始化。20.#加入这个判断的原因是：有时候我们可以用训练好的参数来初始化W，见我的上一篇文章。21.ifWisNone:22.W_values=(3.(24.low=(6./(n_in+n_out)),25.high=(6./(n_in+n_out)),26.size=(n_in,n_out)30.W_values*=431.W=(value=W_values,name='W',borrow=True)2.ifbisNone:33.b=(value=b_values,name='b',borrow=True)37.#隐含层的输出38.lin_output=(input,+40.lin_outputifactivationisNone41.elseactivation(lin_output)42.)43.#隐含层的参数]过来：[python]2.定义分类层LogisticRegression，也即Softmax回归3.在deeplearningtutorial中，直接将LogisticRegression视为Softmax，4.而我们所认识的二类别的逻辑回归就是当n_out=2时的LogisticRegression6.#参数说明：inputnexamplenin)，其中n_example是一个batch的大小，8.#因为我们训练时用的是MinibatchSGD，因此input这样定义9.#n_in,即上一层(隐含层)的输出10.#n_out,输出的类别数11.classLogisticRegression(object):12.def__init__(self,input,n_in,n_out):value=(16.(n_in,n_out),19.borrow=Truevalue(23.(n_out,),26.borrow=Truetnexamplenout29.#故p_y_given_x每一行代表每一个样本被估计为各类别的概率mplenoutnexampleb，b31.#然后(n_example,n_out)矩阵的每一行都加b34.=,axis=1)NLeNet5(当然，是简化的，上面已经说了)，具体来说，就是组装成：NetinputLeNetConvPoolLayerLeNetConvPoolLayer (3)加载MNIST数据集()[python]2.加载MNIST数据集load_data()4.defload_data(dataset):5.#dataset是数据集的路径，程序首先检测该路径下有没有MNIST数据集，没有的话就下载6.#这一部分就不解释了，与softmax回归算法无关。7.ifdata_dir==""andnot8.#Checkifdatasetisinthedatadirectory.9.0],10."..",11."data",dataset)14.ifordata_file==:15.dataset=new_path16.if(notanddata_file==:17.importurllib18.origin=()20.print'Downloadingdatafrom%s'%origin21.(origin,dataset)22.print'...loadingdata'24.#‘rb’表示以二进制可读的方式打开文件26.train_set,valid_set,test_set=(f)sharedvariables，主要时为了GPU加速，只有sharedvariables才能存到GPUmemory中int30.defshared_dataset(data_xy,borrow=True):31.data_x,data_y=data_xy32.shared_x=(data_x,33.borrow=borrow)34.shared_y=(data_y,35.borrow=borrow)37.test_set_x,test_set_y=shared_dataset(test_set)38.valid_set_x,valid_set_y=shared_dataset(valid_set)39.train_set_x,train_set_y=shared_dataset(train_set)40.rval=[(train_set_x,train_set_y),(valid_set_x,valid_set_y),41.(test_set_x,test_set_y)]42.returnrval (4)实现LeNet5并测试[python]2.实现LeNet53.LeNet5有两个卷积层，第一个卷积层有20个卷积核，第二个卷积层有50个卷积核"5.defevaluate_lenet5(learning_rate=,n_epochs=200,6.dataset=,7.nkerns=[20,50],batch_size=500):9.learning_rate:学习速率，随机梯度前的系数。nepochs会遍历所有batch，即所有样本11.batch_size,这里设置为500，即每遍历完500个样本，才计算梯度并更新参数12.nkerns=[20,50],每一个LeNetConvPoolLayer卷积核的个数，第一个LeNetConvPoolLayer有)16.#加载数据17.datasets=load_data(dataset)18.train_set_x,train_set_y=datasets[0]19.valid_set_x,valid_set_y=datasets[1]20.test_set_x,test_set_y=datasets[2]22.n_train_batches=(borrow=True).shape[0]23.n_valid_batches=(borrow=True).shape[0]24.n_test_batches=(borrow=True).shape[0]25.n_train_batches/=batch_size26.n_valid_batches/=batch_size27.n_test_batches/=batch_size29.index=()x('x')y('y')32.######################33.#BUILDACTUALMODEL#34.######################35.print'...buildingthemodel'36.#我们加载进来的batch大小的数据是(batch_size,28*28)，但是LeNetConvPoolLayer的输入是四维的，所以要reshape37.layer0_input=((batch_size,1,28,28))LeNetConvPoolLayer40.#经过maxpooling得到(24/2,24/2)=(12,12)41.#因为每个batch有batch_size张图，第一个LeNetConvPoolLayer层有nkerns[0]个卷积核，42.#故layer0输出为(batch_size,nkerns[0],12,12)43.layer0=LeNetConvPoolLayer(rng45.input=layer0_input,46.image_shape=(batch_size,1,28,28),47.filter_shape=(nkerns[0],1,5,5),48.poolsize=(2,2)49.)NetConvPoolLayer51.#输入是layer0的输出，每张特征图为(12,12),经过conv得到(12-5+1,12-5+1)=(8,8),52.#经过maxpooling得到(8/2,8/2)=(4,4)53.#因为每个batch有batch_size张图(特征图)，第二个LeNetConvPoolLayer层有nkerns[1]个卷积核55.layer1=LeNetConvPoolLayer(ut58.image_shape=(batch_size,nkerns[0],12,12),#输入nkerns[0]张特征图，即layer0输出nkerns[0]张特征图59.filter_shape=(nkerns[1],nkerns[0],5,5),60.poolsize=(2,2))是一个全连接层，相当于MLP里面的隐含层yer(batch_size,num_pixels)，64.#故要将上层中同一张图经不同卷积核卷积出来的特征图合并为一维向量，65.#也就是将layer1的输出(batch_size,nkerns[1],4,4)flatten为nkernslayer(batch_size,n_out)=(500,500)68.layer2=HiddenLayer(70.input=layer2_input,71.n_in=nkerns[1]*4*4,72.n_out=500,73.activation=.)75.#最后一层layer3是分类层，用的是逻辑回归中定义的LogisticRegression，(batch_size,n_out)=(500,10)77.layer3=LogisticRegression(input=,n_in=500,n_out=10)78.#代价函数NLL79.cost=(y)构，83.test_model=(84.[index],86.givens={87.x:test_set_x[index*batch_size:(index+1)*batch_size],88.y:test_set_y[index*batch_size:(index+1)*batch_size]90.)92.validate_model=(93.[index],95.givens={96.x:valid_set_x[index*batch_size:(index+1)*batch_size],97.y:valid_set_y[index*batch_size:(index+1)*batch_size]}99.)100.#下面是train_model，涉及到优化算法即SGD，需要计算梯度、更新参数101.#参数集102.params=+++103.#对各个参数的梯度104.grads=(cost,params)105.#因为参数太多，在updates规则里面一个一个具体地写出来是很麻烦的，所以下面用了一个for..in..,自动生成规则对(param_i,param_i-learning_rate*grad_i)106.updates=[107.(param_i,param_i-learning_rate*grad_i)108.forparam_i,grad_iinzip(params,grads)109.]110.#train_model，代码分析同test_model。train_model里比test_model、validationmodelupdates规则111.train_model=(112.[index],113.cost,114.updates=updates,115.givens={116.x:train_set_x[index*batch_size:(index+1)*batch_size],117.y:train_set_y[index*batch_size:(index+1)*batch_size]118.}119.)120.###############121.#开始训练#122.###############123.print'...training'124.patience=10000125.patience_increase=2126.improvement_threshold=127.validation_frequency=min(n_train_batches,patience/2)128.#这样设置validation_frequency可以保证每一次epoch都会在验证集上测试。129.best_validation_loss=#最好的验证集上的loss，最好即最小130.best_iter=0#最好的迭代次数，以batch为单位。比如best_iter=10000，说明在训练完第10000个batch时，达到best_validation_loss131.test_score=0.132.start_time=()133.epoch=0134.done_looping=False135.#下面就是训练过程了，while循环控制的时步数epoch，一个epoch会遍历所有的train_model(minibatch_index)去训练模型，137.#train_model里面的updatas会更新各个参数。nfrequency数时则会在验证集上测试，139.#如果验证集的损失this_validation_loss小于之前最佳的损失best_validation_loss，140.#则更新best_validation_loss和best_iter，同时在testset上测试。141.#如果验证集的损失this_validation_loss小于best_validation_loss*improvement_threshold时则更新patience。142.#当达到最大步数n_epoch时，或者patience<iter时，结束训练143.while(epoch<n_epochs)and(notdone_looping):144.epoch=epoch+1145.forminibatch_indexinxrange(n_train_batches):146.iter=(epoch-1)*n_train_batches+minibatch_index147.ifiter%100==0:148.print'training@iter=',iter149.cost_ij=train_model(minibatch_index)150.#cost_ij没什么用，后面都没有用到,只是为了调用train_model，而train_model有返回值151.if(iter+1)%validation_frequency==0:152.#computezero-onelossonvalidationset153.validation_losses=[validate_model(i)fori154.inxrange(n_valid_batches)]155.this_validation_loss=(validation_losses)156.print('epoch%i,minibatch%i/%i,validationerror%f%%'%157.(epoch,minibatch_index+1,n_train_batches,158.this_validation_loss*100.))159.ifthis_validation_loss<best_validation_loss:160.ifthis_validation_loss<best_validation_loss*\161.improvement_threshold:162.patience=max(patience,iter*patience_increase)163.best_validation_loss=this_validation_loss164.best_iter=iter165.test_losses=[166.test_model(i)167.foriinxrange(n_test_batches)168.]169.test_score=(test_losses)170.print(('epoch%i,minibatch%i/%i,testerrorof'171.'bestmodel%f%%')%172.(epoch,minibatch_index+1,n_train_batches,173.test_score*100.))174.ifpatience<=iter:175.done_looping=True176.break177.end_time=()179.print('Bestvalidationscoreof%f%%obtainedatiteration%i,'180.'withtestperformance%f%%'%181.(best_validation_loss*100.,best_iter+1,test_score*100.))182.print>>,('Thecodeforfile'+183.1]+184.'ranfor%.2fm'%((end_time-start_time)/60.))TheConvolutionOperatorutionallayerinTheanoymbolicinputs•a4Dtensorcorrespondingtoamini-batchofinputimages.Theshapeofutfeaturemapseightimagewidth•a4Dtensorcorrespondingtotheweightmatrix.TheshapeofthetensoreldsimportimporttheanofromtheanoimporttensorasTfromimportconv2dimportnumpyrng=instantiate4Dtensorforinputinput=(name='input')#initializesharedvariableforweights.w_shp=(2,3,9,9)ww_bound=(3*9*9)W=(((low=/w_bound,high=/w_bound,size=w_shp),#initializesharedvariableforbias(1Dtensor)withrandomvalues#IMPORTANT:biasesareusuallyinitializedtozero.Howeverinthis#particularapplication,wesimplyapplytheconvolutionallayerto#animagewithoutlearningtheparameters.Wethereforeinitialize#themtorandomvaluesto"simulate"learning.b_shp=(2,)b=((low=,high=.5,size=b_shp),#buildsymbolicexpressionthatcomputestheconvolutionofinputwithfiltersinwconv_out=conv2d(input,W)#buildsymbolicexpressiontoaddbiasandapplyactivationfunction,.produceneuralnetlayeroutput#Afewwordson``dimshuffle``:#``dimshuffle``isapowerfultoolinreshapingatensor;#whatitallowsyoutodoistoshuffledimensionaround#butalsotoinsertnewonesalongwhichthetensorwillbe#broadcastable;#dimshuffle('x',2,'x',0,1)#Thiswillworkon3dtensorswithnobroadcastable#dimensions.Thefirstdimensionwillbebroadcastable,#thenwewillhavethethirddimensionoftheinputtensoras#thesecondoftheresultingtensor,etc.Ifthetensorhas#shape(20,30,40),theresultingtensorwillhavedimensions#(1,40,1,20,30).(AxBxCtensorismappedto1xCx1xAxBtensor)#Moreexamples:#dimshuffle('x')->makea0d(scalar)intoa1dvector#dimshuffle(0,1)->identity#dimshuffle(1,0)->invertsthefirstandseconddimensions#dimshuffle('x',0)->makearowoutofa1dvector(Nto1xN)#dimshuffle(0,'x')->makeacolumnoutofa1dvector(NtoNx1)#dimshuffle(2,0,1)->AxBxCtoCxAxB#dimshuffle(0,'x',1)->AxBtoAx1xB#dimshuffle(1,'x',0)->AxBtoBx1xAoutput=+('x',0,'x','x'))#createtheanofunctiontocomputefilteredimagesff=([input],output)fromfromimportpoolmaxpool_shape=(2,2)pool_out=(input,maxpool_shape,ignore_border=True)f=([input],pool_out)invals=2,5,5)print'Withignore_bordersettoTrue:'print'invals[0,0,:,:]=\n',invals[0,0,:,:]print'output[0,0,:,:]=\n',f(invals)[0,0,:,:]pool_out=(input,maxpool_shape,ignore_border=False)f=([input],pool_out)print'Withignore_bordersettoFalse:'print'invals[1,0,:,:]=\n',invals[1,0,:,:]print'output[1,0,:,:]=\n',f(invals)[1,0,:,:]TheFullModel:LeNet：classLeNetConvPoolLayer(object):"""PoolLayerofaconvolutionalnetwork"""def__init__(self,rng,input,filter_shape,image_shape,poolsize=(2,2)):AllocateaLeNetConvPoolLayerwithsharedvariableinternalparameters.:paramrng:arandomnumbergeneratorusedtoinitializeweights:paraminput:symbolicimagetensor,ofshapeimage_shape::typefilter_shape:tupleorlistoflength4:paramfilter_shape:(numberoffilters,numinputfeaturemaps,filterheight,filterwidth):typeimage_shape:tupleorlistoflength4:paramimage_shape:(batchsize,numinputfeaturemaps,imageheight,imagewidth):typepoolsize:tupleorlistoflength2:parampoolsize:thedownsampling(pooling)factor(#rows,#cols)assertimage_shape[1]==filter_shape[1]=input#thereare"numinputfeaturemaps*filterheight*filterwidth"#inputstoeachhiddenunitfan_in=(filter_shape[1:])#eachunitinthelowerlayerreceivesagradientfrom:#"numoutputfeaturemaps*filterheight*filterwidth"/#poolingsizefan_out=(filter_shape[0]*(filter_shape[2:])/(fan_in+fan_out))=(((low=-W_bound,high=W_bound,size=filter_shape),dtype=),borrow=True)#thebiasisa1Dtensor--onebiasperoutputfeaturemapb_values=((filter_shape[0],),dtype==(value=b_values,borrow=True)#convolveinputfeaturemapswithfiltersconv_out=conv2d(input=input,filters=,filter_shape=filter_shape,input_shape=image_shape)#pooleachfeaturemapindividually,usingmaxpoolingpooled_out=(input=conv_out,ds=poolsize,ignore_border=True)#addthebiasterm.Sincethebiasisavector(1Darray),wefirst#reshapeittoatensorofshape(1,n_filters,1,1).Eachbiaswill#thusbebroadcastedacrossmini-batchesandfeaturemap#width&height=(pooled_out+'x',0,'x','x'))#storeparametersofthislayer=[,]#keeptrackofmodelinput=input注意，在初始化权重值时，扇入取决于接收字段的大小和输入特征映射的数量。x=('x')#thedataispresentedasrasterizedimagesy=('y')#thelabelsarepresentedas1Dvectorof#[int]labels#######################BUILDACTUALMODEL#######################print('...buildingthemodel')#Reshapematrixofrasterizedimagesofshape(batch_size,28*28)#toa4Dtensor,compatiblewithourLeNetConvPoolLayer#(28,28)isthesizeofMNISTimages.layer0_input=((batch_size,1,28,28))#Constructthefirstconvolutionalpoolinglayer:#filteringreducestheimagesizeto(28-5+1,28-5+1)=(24,24)#maxpoolingreducesthisfurtherto(24/2,24/2)=(12,12)#4Doutputtensoristhusofshape(batch_size,nkerns[0],12,12)layer0=LeNetConvPoolLayer(input=layer0_in