神经网络课件

智能系统(AComprehensiveFoundation)LectureNoteZhaoJieYuComputerScienceDepartmentNingboUniversityAComprehensiveFoundationReferences:NeuralNetworksandLearningMachines---Acomprehensivefoundation(3rdEdition),SimonHaykin,PrenticeHall,20092.InformationTheory,Inference,andLearningAlgorithms,DavidJ.C.MacKay,CambridgeUniversityPress,20033.智能科学,史忠植,清华大学出版社,20064.WiKi,/Contents:NeuralComputation,LearningMachinesSupportVectorMachinePrincipalComponentsAnalysisRecurrentNetworks,MarkovRandomFieldsEvolutionaryComputationPart1.NeuralComputationWhatisanArtificialNeuralNetwork?ComputationalmodelinspiredfromneurologicalmodelofbrainHumanbraincomputesindifferentwayfromdigitalcomputer(thevonNeumannmachine)highlycomplex,nonlinear,andparallelcomputingmanytimesfasterthand-computerinpatternrecognition,perception,motorcontrolhasgreatstructureandabilitytobuildupitsownrulesbyexperiencedramaticdevelopmentwithin2yearsafterbirthcontinuestodevelopafterwardLanguageLearningDevicebefore13yearsoldPlasticity:abilitytoadapttoitsenvironmentThemostimportant:itisalive!HistoricalNotesDeepLearning…….Markovrandomfields,probabilisticgraphicalmodelsspikingneuralnetworksVapnik(1990)---SupportVectorMachine(SVM)Broomhead&Lowe(1988)----RadialBasisFunctions(RBF)Linsker(1988)-----InformaxprincipleRumelhart,Hinton(1986)--------Back-propagation[Werbos(1974)-----Beyondregression(PhDThesis)]Kohonen(1982)------Self-OrganizingMapsHopfield(1982)------HopfieldNetworksMinsky&Papert(1969)------PerceptronsRosenblatt(1960)------PerceptronMinsky(1954)------NeuralNetworks(PhDThesis)Hebb(1949)--------TheorganizationofbehaviourMcCulloch&Pitts(1943)-----artificialneuralnetworksbornNeuralNetworkDefinitionsMachinedesignedtomodelthewayinwhichbrainperformstasksimplementedbyelectronicdevicesand/orsoftware(simulation)LearningisthemajoremphasisofNNMassivelyparalleldistributedprocessormassiveinterconnectionofsimpleprocessingunitssimpleprocessingunitsstoreexperienceandmakeitavailabletouseknowledgeisacquiredfromenvironmentthrulearningprocessLearningMachinemodifysynapticweightstoobtaindesignobjectivemodifyowntopology-neuronsdieandnewonecangrowConnectionistnetwork-connectionismBenefitsofNeuralNetworks(I)Powercomesfrommassivelyparalleldistributedstructureandlearntogeneralizegeneralization:abilitytoproducereasonableoutputforinputsnotencounteredduringtrainingNNcannotprovidesolutionbyworkingindividuallyComplexproblemisdecomposedintosimpletasks,andeachtaskisassignedtoaNNLongwaytogotobuildacomputerthatmimichumanbrain1.Non-linearityinterconnectionofnon-linearneuronsisitselfnon-lineardesirablepropertyifunderlyingphysicalmechanismisnon-linearBenefitsofNeuralNetworks(II)2.Input-OutputMappinginput-outputmappingisbuiltbylearningfromexamplesreducedifferencesofdesiredresponseandactualresponsenon-parametricstatisticalinferenceestimatearbitrarydecisionboundariesininputsignalspace3.AdaptivityadaptsynapticweighttochangesofenvironmentNNisretrainedtodealwithminorchangeintheoperatingenvironmentchangesynapticweightsinreal-timemorerobust,reliablebehaviorinnon-stationaryenvironmentAdaptivepatternrecognition,Adaptivesignalprocessing,Adaptivecontrolstability-plasticitydilemmaBenefitsofNeuralNetworks(III)4.EvidentialResponsenotonlyselectedclasslabelbutalsoconfidenceconfidencescanbeusedtorejectrecognitionaccuracyvs.reliability(doonlyyoucando)5.ContextualInformationprocessing(contextual)knowledgeispresentedinthestructureeveryneuronisaffectedbyothers6.FaultToleranceperformancedegradesgracefullyunderadverseconditioncatastrophicfailureofd-computer7.VLSIimplementabilitymassivelyparallelnaturemakesitwellsuitedforVLSIimplementationBenefitsofNeuralNetworks(IV)8.UniformityofAnalysisandDesignNeuroniscommontoallNNsharetheoriesandlearningalgorithmsmodularnetworkscanbebuiltthruseamlessintegration9.NeurobiologicalAnalogylivingproofoffaulttolerant,fast,powerfulprocessingNeuroscientistsseeitasaresearchtoolforneurobiologicalphenomenaEngineerslooktoneurosciencefornewideasHumanBrainHumannervoussystem1011neuronsinhumancortex60x1012synapticconnections104synapsesperneuron10-3seccycletime(computer:10-9sec)energeticefficiency:10-16joulesoperationpersecond(computer:10-6joules)ReceptorsNeuralNetEffectorsstimulusresponse核磁共振成像HumanBrainNeuronstructurenucleus,cellbody,axon,dendrite,synapsesNeurotransmissionNeuron’soutputisencodedasaseriesofvoltagepulsescalledactionpotentialsorspikesbymeansofelectricalimpulseeffectedbychemicaltransmitterPeriodoflatentsummationgenerateimpulseiftotalpotentialofmembranereachesalevel:firingexcitatoryorinhibitoryCerebralcortexAreasofBrainforspecificfunctionBiologicalNeuronExcerptedfromArtificialIntelligence:ModernApproach

byS.RusselandP.NorvigNeurotransmission/pubs/qdocs/mom/TG/intro.htmOrganizationofLevelsinBrainsmapintocerebralcortexpathways,columns,topographicmaps;involvemultipleregionsneuronsofsimilaranddifferentproperties,1mminsize,localizedregioninthebrain100minsize,containsseveraldendritetreesAssemblyofsynapses(siliconchip),milisecond,mmostfundamentallevel(transistor)MoleculesSynapsesNeuralmicrocircuitsDendritetreeNeuronsLocalcircuitsInterregionalcircuitsCentralNervoussysCerebral

CortexModelsofNeuronNeuronisinformationprocessingunitAsetofsynapsesorconnectinglinkscharacterizedbyweightorstrengthAnaddersummingtheinputsignalsweightedbysynapsesalinearcombinerAnactivationfunctionalsocalledsquashingfunctionsquash(limits)theoutputtosomefinitevaluesNonlinearmodelofaneuron(I)wk1x1wk2x2wkmxm......

Biasbk

(.)vkInputsignalSynapticweightsSummingjunctionActivationfunctionOutputykNonlinearmodelofaneuron(II)wk1x1wk2x2wkmxm......

(.)vkInputsignalSynapticweightsSummingjunctionActivationfunctionOutputykwk0X0=+1Wk0=bk(bias)TypesofActivationFunctioniniOj+1iniOj+1t阈值函数Piecewise-linearFunctionSigmoidFunction(differentiable)iniOj+1taisslopeparameterActivationFunctionvaluerangevi+1-1SignumFunctionvi+1HyperbolictangentFunctionStochasticModelofaNeuronDeterministicvsstochasticstochastic:stayatastatewithprobabilityx:stateofneuronv:inducedlocalfieldP(v)probabilityoffiringwhereTispseudotemparatureT0,reducedtodeterministicformNeuralnetworkasdirectedGraphBlockdiagramcanbesimplifybytheideaofsignalflowgraphnodeisassociatedwithsignaldirectedlinkisassociatedwithtransferfunctionsynapticlinksgovernedbylinearinput-outputrelationsignalxjismultipliedbysynapticweightwkjactivationlinksgovernedbynonlinearinput-outputrelationnonlinearactivationfunctionSignalFlowGraphofaNeuronx1x2xm...vkykx0=+1

(.)Wk0=bkwk1wk2wkmArchitecturalgraphofaNeuronPartiallycompletedirectedgraphdescribinglayoutComputationnode:shadedsourcenode:smallsquareThreegraphicalrepresentationsBlockdiagram-providingfunctionaldescriptionofaNNSignalflowgraph-completedescriptionofsignalflowarchitecturalgraph-networklayoutFeedback输出部分决定自己的输出通过反馈dependingonw稳定，线性发散，发散指数weareinterestedinthecaseof|w|<1;infinitememoryoutputdependsoninputsofinfinitepastNNwithfeedbackloop:recurrentnetworkxj(n)xj’(n)wyk(n)z-1NetworkArchitectureSingle-layerFeedforwardNetworksinputlayerandoutputlayersingle(computation)layerfeedforward,acyclicMultilayerFeedforwardNetworkshiddenlayers-hiddenneuronsandhiddenunitsenablestoextracthighorderstatistics10-4-2network,100-30-10-3networkfullyconnectedlayerednetworkRecurrentNetworksatleastonefeedbackloopwithorwithouthiddenneuronNetworkArchitectureSinglelayerMultiplelayerfullyconnectedRecurrentnetworkwithouthiddenunitsinputsoutputsRecurrentnetworkwithhiddenunitsUnitdelayoperatorKnowledgeRepresentationKnowledgereferstostoredinformationormodelsusedbyapersonormachinetointerpret,predictandappropriatelyrespondtotheoutsideworldWhatinformationisactuallymadeexplicit;HowtheinformationisphysicallyencodedforthesubsequentuseGoodsolutiondependsongoodrepresentationofknowledgeInNN,knowledgeisrepresentedbyinternalnetworkparametersrealchallengeKnowledgeoftheworldworldstaterepresentedbyknownfacts-priorknowledgeobservations-obtainedby(noisy)sensors;trainingexamplesKnowledgeAcquisitionbyNNTrainingTrainingexamples:eitherlabeledorunlabeledlabeled:inputsignalanddesiredresponseunlabeled:differentrealizationsofinputsignalExamplesrepresenttheknowledgeofenvironmentHandwrittendigitrecognition1.AppropriatearchitectureisselectedforNNsourcenode=numberofpixelsofinputimage10outputnodeforeachdigitsubsetofexamplesfortrainingNNbysuitablelearningalgorithm2.RecognitionperformanceistestedbytherestoftheexamplesPositiveandnegativeexamplesRulesofKnwldgrepresentationinNN1.SimilarinputfromsimilarclassesproducesimilarrepresentationssimilaritymeasuresEuclidiandistance,dot(inner)product,cosrandomvariable:Mahalanobisdistance...2.Separateclassesproducewidelydifferentrepresentations3.Moreneuronsshouldbeinvolvedinrepresentationofmoreimportantfeatureprobabilityofdetection/falsealarm4.PriorinformationandinvariancesshouldbebuiltintothedesignofthenetworkgeneralpurposevsspecializedBuildingPriortoNNdesignSpecializedstructurelearnsfastbecauseofsmallfreeparametersrunsfastbecauseofsimplestructureNowell-definedrulesforbuildingspecializedNNadhocapproeachrestrictingthenetworkarchitecturethruuseoflocalconnectionreceptivefieldConstrainingthechoiceofsynapticweightsweightsharing,parametertyingx1x5x6x7x8x9x10x3x4x2y1y2Combiningreceptivefieldandweight-sharing(convolutionnetwork)BuildinginvariancetoNNdesignWanttobecapabletocopewithtransformationsInvariancebystructuresynapticconnectionsarearrangednottoaffectedbythetransformrotationinvariantforcingwji

=wjkforallkinthesamedistancefromthecenterofimageInvariancebytrainingtrainbydataofmanydifferenttransformationscomputationallyinfeasibleinvariantfeaturespaceusefeaturesinvarianttothetransformationsNowell-developedtheoryofoptimizingarchitectureofNNNNlacksexplanationcapabilityAIandNNDefinitionofAI;GoalofAIartofcreatingmachinethatperformstasksthatrequiresintelligencewhenperformedbypeoplestudyofmentalfacultiesthroughtheuseofcomputat