第2章信源与信源熵

1、2022-6-112022-6-122022-6-132022-6-142022-6-15信源无记忆信源有记忆信源发出单个符号的无记忆信源发出符号序列的无记忆信源发出符号序列的有记忆信源发出符号序列的马尔可夫信源2022-6-162022-6-1712,nXa aa12,nPp ap ap a1212nnaaaXp ap ap aP10,1niiip ap a2022-6-182022-6-190( )()(/)(/)()1ijjiijijp xp yp yxp xyp x y，11111111()1,()1,(/)1,(/)1,()1,()(),()()nmnijijijimmnjiijjj

2、inmijjijiijp xp yp xyp yxp x yp x yp yp x yp x2022-6-110()() (/)() (/)ijijijijp x yp x p yxp yp xy()( ) ()(/)()(/)( )ijijjijijiXYp x yp x p yp yxp yp xyp x当 与 相互独立时，11()()(/)(/)()()ijijijjinmijijijp x yp x yp xyp yxp x yp x y，2022-6-111123456111111666666aaaaaaXP610,1iiip ap a2022-6-1121212nnaaaXp ap

3、 ap aP10,1niiip ap a2022-6-113( , )( )Xa bXpxP( )0( )1XbXapxpx dx 2022-6-11412,lLXXXXX12,1,2,lnXa aalL001010011 100101 110111111666666XP123,3,0,1lXXXXLX2022-6-1151211121112,nnnnXXa aa aa aXp a ap a ap a aPX ,/ijijiijp a ap ap aap ap a2022-6-1160101XppP0001101100011011XppppP00000101001110010111011100

4、0001010011100101110111XppppppppP000102101112202122000102101112202122XpppppppppP2022-6-1171212LLnLnXpppP12iiiiLX XX2022-6-1182022-6-1192022-6-120 12112121/LLLLXXXXpXXXpXXXp 1212211221/LLLLLXXXXpXXXXpXXXp 121121/LLp X p XXp XX XX 2022-6-1212022-6-1221211(/)(/)LLLL mLp XX XXp XXX123121312122112132121()

5、() (/) (/)(/)() (/) (/)(/) (/)LLLLLLLLp X X XXp X p XX p XX Xp XX XXXp X p XX p XXp XXp XX2022-6-12312lLX XXXX2022-6-1242022-6-1252022-6-1262022-6-1272022-6-128( ) ( )iiI xf p x 1ixp ( )0if p x0ip x ( )if p x 2022-6-129 iiixpxpxIlog1log2022-6-1302022-6-131mxxnnmlogloglog1021010loglog3.322loglog2xxx2

6、022-6-1322022-6-133o 例：例：英文字母中英文字母中“e” 出现的概率为出现的概率为0.105，“c”出现的概率为出现的概率为0.023，“z”出现的概率为出现的概率为0.001。分别计算它们的自信息量。分别计算它们的自信息量。o 解：解：“e”的自信息量的自信息量 I(e) =log20.105=3.25 bit “c”的自信息量的自信息量 I(c) =log20.023=5.44 bit “z”的自信息量的自信息量 I(z) =log20.001=9.97 bit2022-6-1342022-6-135)(log)(jijiyxpyxI2022-6-136(/)log(/

7、)ijijI xyp xy 22()log() (/)()(/)log() (/)()(/)ijijiijijijjijI x yp x p yxI xI yxp yp xyI yI xy 2022-6-137 bit322. 08 . 0loglog2121xpxI120.80.2XxxP2222loglog 0.22.322bitI xp x 2022-6-138 2211xIxnpxIxnp 11221H Xnp xI xnp xI xn121222loglogp xp xp xp x 221log0.722 /iiip xp x 比特 次2022-6-139()()( ) ( )( )

8、log( )iiiiiiH XE I Xp x I xp xp x 2022-6-140120.990.01XxxP120.50.5YyyP( )0.99log0.99 0.01log0.01 0.08 /H X 比特 符号( )0.5log0.50.5log0.51 /H Y 比特 符号2022-6-14153 1010n5500 6003 10 53 1022156()()log()log 10 =3.322 3 1010 /niiiH Xp xp x 比特 画面2022-6-142342()log3.322 4 101.3 10 /H XN 比特 千字文2022-6-14301XPpq(

9、)- log- log - log-(1- )log(1- ) ( )H Xpp qqppppH p2022-6-1442022-6-1452,/() (/)()log(/)ijjiijjii ji jH YXp x yI yxp x yp yx ()/H X Y()2(/)(/) (/)(/)log(/)jijijijijiiH Xyp xy I xyp xyp xy 2,() (/)()log(/)ijijijiji ji jp x yI xyp x yp xy ()(/)jjjp yH Xy=() (/) (/)jijijijp yp xyI xy2022-6-146 112,0,0/l

10、og/ijijijiji ji jH X Yp x yI xyp x yp xy 130,01,1,0,11,088p XYp XYp XYp XY10/ijijijip x yp xyp x y0000010110101111log/log/log/log/p x yp xyp x yp xyp x yp xyp x yp xy 0112p yp y222211333311loglogloglog84848484 0.811 /比特 符号,/()(/)() (/) (/)jjjijijji jH X Yp yH Xyp yp xyI xy()=2022-6-1472,()() ()()log

11、()ijijijiji ji jH XYp x yI x yp x yp x y /H XYH XH YXH XYH YH X Y2022-6-148/H XYH XH YX,()() ()()log()()log( ) (/)ijijijiji ji jijijii jH XYp x yI x yp x yp x yp x yp x p yx ,()log( )()log(/)ijiijjii ji jp x yp xp x yp yx ,log( )()log()(/)iijjiiijji jp xp x yp x yp yx ,( )log( )()log(/)()( /)iiijjii

12、i jp xp xp x yp yxH XH YX 2022-6-149p(y1)p(x4)111111117loglogloglog224488884H X 114log244H Y 1111111127log2log6log4log4488161632328H XY 2022-6-1502022-6-151000002001021022( /)( /)log( /)( /)log( /)( /)log( /)3311loglog0.82/4444iiiH V up v up v up v up v up v up v ubit 符号2022-6/)()log(/)

13、33111111loglogloglog848444440.91 /ijjiijH V Up u vp vu 比特 符号1100()()log()1.91 /ijijijH UVp u vp u v 比特 符号 /H UVH UH V U 1111loglog1 /2222H U 比特 符号1.91 /比特 符号2022-6-1533 5( )(,)0.96/8 8(/)()( )1.91 0.960.95/H VHbitH U VH UVH Vbit符号符号1100(/)()log(/)0.95/ijijijH U Vp u vp uvbit 符号2022-6-1542022-6-15522

14、11()loglog0.92/3333H Xbit 符号)/()()/()()(jijijijiyxpypxypxpyxp( / )()log(/ )13111111loglogloglog0.88/24646262ijjiijH Y Xp x yp y xbit 符号2022-6-156(/ )()log (/)0.33/ijijijH X Yp x yp xybit符号10100()(/)0()p x yp xyp y00000()1/ 2(/)1()1/ 2p x yp xyp y1()()nijjip x yp y111111( )logloglog1.47/223366H Ybit

15、符号2022-6-1572022-6-158有扰信道干扰源信源X信宿Y)()()(2121nnxpxpxpxxxPX)()()(2121nnypypypyyyPY2022-6-1592022-6-1602(/)log( )ijip xyp x( ;)( )(/)ijiijI x yI xI xy2022-6-1611/ 21/ 41/81/8XP晴阴雨雪2022-6-1622022-6-16342421/0log2 41/4;0log1 1/8I xbitI xbit 42421/01log1 21/2;01log2 1/8I xbitI xbit 4242/011log 10 1;011lo

16、g3 1/8I xbitI xbit43 I xbit2022-6-164( /)( )( ;) loglog0( )( )ijiijiip x yp xI x yp xp x( ;)(;)ijjiI x yI yx(/)(/)loglog( )()ijjiijp xyp yxp xp y iip xp xjjp yp y2022-6-165(/)(/) ( ;)(/)log(;)jijijijijiiip xyp xyI x yI X yp xyp x,(/)() (;)() (/)log( )(/)()log( )(; )ijjjjijji jiijiji jip xyp y I X yp

17、 yp xyp xp xyp x yp xI X Y,logl/og/iiijijii jp xp xp x ypHXxYyXH 2022-6-1662022-6-167 符号比特/696. 01613,163 HXH 符号比特/0906. 0log;,2yxjijijiypxpyxpyxpYXI2022-6-168 ;/;/I X YH XH X YI Y XH YH YX2022-6-1692022-6-170( ;);( ;/)ijiijkkkI x y zI x zI x yz(/)log( )( ;)ijkiijkp xy zp xI x y z(/)log(/)( ;/)ijkik

18、ijkp xy zp xzI x yz(/)(/)ikikp xzp xz2022-6-171(; )(;/)(;)I X YI X Z YI X YZ(;)(;/)(;)I X ZI X Y ZI X YZ( ;)( ;/)(;)I Y XI Z X YI YZ X2022-6-172XYZ(; )(;/)(;)I X YI X Z YI X YZ(;)(;/)(;)I X ZI X Y ZI X YZ2022-6-173ZYXIYZXIYXIZXI/;/;( ;)( ;/)(;)I Y XI Z X YI YZ X(;)( ;/)(; )I X ZI Y Z XI XY Z( ;)(;/)

19、(; )I Y ZI X Z YI XY Z;/;/I X ZI Y ZI X Z YI Y Z X;I X ZI X Y;I X ZI Y Z2022-6-174;I U VI X Y;I U VI X VI X VI X Y2022-6-175123111326xxxXP123111362yyyYP123111236zzzZP H XH YH Z2022-6-17612120lim,nnH p ppH p pp 2022-6-177111,logH XHMMMM(/)()(/)(/)()()( )H X YH XH ZXYH Z YH XYH XH Y2022-6-178( ; )( )(

20、 / )( )( / )( )( )()I X YH XH X YH YH Y XH XH YH XY()( )( /)( )(/ )()( )( )( )(/ )( )( /)H XYH XH Y XH YH X YH XYH XH YH XH X YH YH Y XH(X/Y)H(X)H(Y)H(XY)H(Y/X)I(X;Y)2022-6-1792022-6-180X1X2XlXL112121312121321111212()(,)() (/) (/)(/)() (/) (/)(/)iLLLLiiiLiiiiiiiiiiiiiiiLiip Xp XxXxXxp xp xxp xx xp x

21、p xp xxp xxxpxxxxx11211LLiiiixx xx2022-6-181121231( )()() () ()()()LLlLiiiiiiiiilpp x xxp xp xp xp xxp x12()()()LH XH XH X1()( )log( )LiniiXxxHpp 111()log()LllLLniiillp xp x 121212111logloglogLLLnnniiiiiiiiip xp xp xp xp xp x 112212111logloglogLLLnnniiiiiiiiip xp xp xp xp xp x 1()LllH X11212111logLL

22、nnniiiiiiip xp xp xp x 2022-6-1821LHXXHL1LllHH XLXXH12LH XH XH XH XH X12Lijijijjp xap xap xap a2022-6-18311()2log1 /22H Xbit 符号11()4log2 /44XHbit 序列211 /2HHHbiXtXX符号2022-6-184123111244xxxXP91log3 bit/iiiHpXp 序列211.5 /2HHbXXitH X符号2022-6-18512()()()LLXHH X XXH X1LHXXHL1LllHH XXH XLH X11(/)LlllH XX121

23、11()(/)(/)LLH XH XXH XXX2022-6-18641943611210aaaPX2022-6-187222100(/)()log(/)0.872 /ijjiijH XXp a ap aabit 符号210()()( )log( )1.543 /iiiH XH Xp ap abit 符号12121/2.415 bit/H X XH XH XX序列21211.207 bit/2HH X XX 符号2022-6-1882H X12121212,LLiiiLihihihLp Xx XxXxp Xx XxXx121231/LLLLH XX XXH XX XX1122/LLH XX X

24、X-1232 /LLH XX XX2123/LLH XX XX21/H XX1H X2022-6-1891/LLH XX11LLLHHXX1LLHHXX121LLHH X XXLX 1211211/LLH XH XXH XX XXL1211/LLLH XX XXL12LLLHH X XXX 121121/LLLH X XXH XX XX111/LLLXLHH XX12LXHHHXX XH0HX 2022-6-190limdefLLXXHH121lim/LLLH XX XX2022-6-1912022-6-1922022-6-193,/ijnjmijipm np SsSsp ss2022-6-1

25、941111(/)QjiQQQppPp sspp1111(/)qjiQQqppPp xspp2022-6-1950;1ijijjpp2022-6-196 klk lijirrjrppp 11kkkijirrjirrjrrpp ppp 12kkkkPPPPPPP2022-6-1972022-6-190.5(/)0.8jiPp xs0/00(1/11)0.81/00(0/11)0.20/01(0/10)1/01(1/10)0.5pppppppp0.80.200000.50.5(/)0.50.500000.20.8jiPp ss2022-6-199,12211Y

26、XYXYTXrYrpqPqp2022-6-11002022-6-11012022-6-110201231/302/30XP01231000XP01231/61/21/61/6XP2022-6-1103iijjiW pW limkijjkpW1jjW 2022-6-11042022-6-1105121234( )( )() 1/21/2() 1/32/3(/)() 1/43/4(0100011011) 1/54/5jiaassp asss5/45/100004/34/13/23/100002/12/1)|(43214321sssssspssssij(1)1/2(0)1/2(0)1/3(1)2/300011110(1)3/4(0)1/4(0)1/5(1)4/52022-6-1106131132243244123411241324113524351ijWWWWWWpWWWWWWWWWWijiWW123436,353564,357WWWW11()(/) ( )iiip ap as p s1316161492353354355735221326364426()(/) ( )2353