模式识别研究生

1、PATTERN RECOGNITIONAND MACHINE LEARNINGCHAPTER 1: INTRODUCTION第一章内容目录模式识别与学习的基本问题多项式曲线拟合实例学习中的基础理论概率论决策论信息论ExampleHandwritten Digit Recognition相关的基本概念训练集合x1,，xN目标向量 t函数: y(x)特征提取预处理推广性(举一反三)监督学习: 分类，回归(regression)无监督学习: 聚类，概率密度估计，可视化reinforcement learning模式识别与学习的基本问题监督学习: 分类，回归(regression)输入变量: x,目标

2、变量: t给定训练样本: x,t目标：（学习出x 和t的函数关系）给定x t模式识别与学习的基本问题学习数据用函数或其它模型表示数据算法：解释数据结果：数据衡量算法：推广能力理论原则：拟合训练数据+最简模型Polynomial Curve Fitting问题描述输入变量: x目标变量: t生成过程: 实际问题中是未知的给定训练样本: x,tPolynomial Curve FittingPolynomial Curve Fitting目标:给定新的x ， 线性模型:的t 值利用训练样本，估计模型的参数方法:误差平方和最小:Sum-of-Squares Error Function0th Ord

3、er Polynomial1st Order Polynomial3rd Order Polynomial9th Order PolynomialPolynomial Curve Fitting哪一个最好？训练误差测试误差过训练均方误差(root-mean-square)Over-fittingRoot-Mean-Square (RMS) Error:Polynomial Curve Fitting过训练的相关因素模型复杂度Polynomial CoefficientsPolynomial Curve Fitting过训练的相关因素模型复杂度训练样本数Data Set Size:9th Ord

4、er PolynomialData Set Size:9th Order PolynomialPolynomial Curve Fitting过训练的相关因素模型复杂度训练样本数学习方法最大似然贝叶斯方法RegularizationPenalize large coefficient valuesRegularization:Regularization:Regularization:vs.Polynomial Coefficients问题思考哪些问题是哪些问题是学习能够解决的？学习不能解决的？哪些问题影响学习的效果？Probability TheoryApples and Oranges P

5、robability TheoryMarginal ProbabilityJoint ProbabilityConditional Probability Probability TheorySum RuleProduct RuleThe Rules of ProbabilitySum RuleProduct RuleBayes Theoremposterior µ likelihood ×priorProbability DensitiesTransformed DensitiesExpectationsConditional Expectation(discrete)A

6、pproximate Expectation(discrete and continuous)Variances and CovariancesThe Gaussian DistributionGaussian Mean and VarianceThe Multivariate GaussianGaussian Parameter EstimationLikelihood functionum (Log) LikelihoodProperties ofandCurve Fitting Re-visitedum LikelihoodDetermineby minimizing sum-of-sq

7、uares error,.Predictive DistributionMAP: A Step towards BayesDetermineby minimizing regularized sum-of-squares error,.Bayesian Curve FittingBayesian Predictive DistributionMSelectionCross-ValidationCurse of DimensionalityCurse of DimensionalityPolynomial curve fitting, M = 3Gaussian Densities inhigh

8、er dimensionsDecision TheoryInference stepDetermine eitheror.Decision stepFor given x, determine optimal t.Minimum Misclassification RateMinimum Expected LossExample: classify medical images as cancer or normalDecisionTruthMinimum Expected LossRegionsare chosen to minimizeReject OptionInference and

9、decisiongenerative msdiscriminative msdiscriminant functionWhy Separate Inference and Decision?Minimizing risk (loss matrix may change over time)Reject option Unbalanced class priorsCombining msDecision Theory for RegressionInference stepDetermine.Decision stepFor given x, make optimal prediction, y

10、(x), for t.Loss function:The Squared Loss FunctionGenerative vs DiscriminativeGenerative approach:MUse Bayes theoremDiscriminative approach:MdirectlyEntropyImportant quantity in coding theory statistical physics machine learningEntropyCoding theory: x discrete with 8 possible states; how manybits to transmit the state of x?All states equally likelyEntropyEntropyIn how many ways can N identical objects be allocated M bins?Entropyized whenEntropyConditional EntropyThe Kullback-Leibler DivergenceMutual InformationContents1 Introduction2 Probability Distributions3 Linear M4 Linear Ms for R