网络故障诊断

网络故障诊断网络故障诊断培训目标掌握一般性故障判断与处理的流程学会使用一至两种工具(软件)来进行故障诊断学会使用系统提供的相关命令来进行网络故障诊断了解一些常见的故障及其可能的原因故障诊断流程故障诊断流程定义故障收集现象估计可能性制订行动计划实施行动观察结果重复过程问题解决,结束现象不存在现象仍存在故障诊断工具故障诊断工具仪表万用表电缆测试仪/光纤测试仪误码表数据发生器协议分析仪故障诊断工具网管工具HPOVSunNetManagerCAUnicenterTNGIBMTivoli/NetViewCiscoWorks其它工具故障诊断工具系统命令Cisco路由器系统命令showdebugpingtracelogging故障诊断工具系统命令Solaris系统命令pingifconfignetstatnslookuparp命令故障诊断工具其它工具snoopsniffertcpdumparpwatch其它软件路由器/交换机的故障诊断正确理解路由器路由与交换的原理了解Cisco路由器硬件结构从路由器诊断命令的输出中发现问题为Cisco技术支持人员提供诊断信息Cisco路由器系统命令Ping命令(user)rtr1-c-1-bjbjTranslating"sun"...domainserver(95)[OK]Typeescapesequencetoabort.Sending5,100-byteICMPEchoesto95,timeoutis2seconds:!!!.!Successrateis80percent(4/5),round-tripmin/avg/max=548/853/1204msPing命令(user)Ping扩展命令r4#pingProtocol[ip]:Repeatcount[5]:10Datagramsize[100]:200Timeoutinseconds[2]:3Extendedcommands[n]:yTypeofservice[0]:1SetDFbitinIPheader?[no]:Validatereplydata?[no]:Datapattern[0xABCD]:0x0000Loose,Strict,Record,Timestamp,Verbose[none]:rNumberofhops[9]:Loose,Strict,Record,Timestamp,Verbose[RV]:Sweeprangeofsizes[n]:Typeescapesequencetoabort.Sending10,200-byteICMPEchosto31,timeoutis3seconds:Packethasdatapattern0x0000PackethasIPoptions:Totaloptionbytes=39,paddedlength=40Ping扩展命令(con’t)Replytorequest0(4ms).ReceivedpackethasoptionsTotaloptionbytes=40,paddedlength=40EndoflistReplytorequest1(4ms).ReceivedpackethasoptionsTotaloptionbytes=40,paddedlength=40EndoflistReplytorequest2(4ms).ReceivedpackethasoptionsTotaloptionbytes=40,paddedlength=40EndoflistSuccessrateis100percent(10/10),round-tripmin/avg/max=1/4/8msTrace命令(user)rtr1-c-1-bjbjTypeescapesequencetoabort.T(0)11()0(2)20(2)20(0)20msec*16msecTrace命令(user)Trace扩展命令r4#traceProtocol[ip]:Numericdisplay[n]:5Timeoutinseconds[3]:Probecount[3]:MinimumTimetoLive[1]:2MaximumTimetoLive[30]:PortNumber[33434]:Loose,Strict,Record,Timestamp,Verbose[none]:rNumberofhops[9]:Loose,Strict,Record,Timestamp,Verbose[RV]:Typeescapesequencetoabort.Trace扩展命令(Con‘t)310msecReceivedpackethasoptionsTotaloptionbytes=40,paddedlength=40Endoflist*0msecReceivedpackethasoptionsTotaloptionbytes=40,paddedlength=40EndoflistShow命令rtr1-c-1-bjbj>shintf4/0/0FastEthernet4/0/0isadministrativelydown,lineprotocolisdownHardwareiscyBusFastEthernetInterface,addressis0050.a253.8080(bia0050.a253.8080)Description:<beijingC1-beijingRegion>100MMTU1500bytes,BW100000Kbit,DLY100usec,rely255/255,load1/255EncapsulationARPA,loopbacknotset,keepaliveset(10sec),fdx,100BaseTX/FXARPtype:ARPA,ARPTimeout04:00:00Lastinput10w6d,output10w6d,outputhangneverLastclearingof"showinterface"counters18w4dQueueingstrategy:fifoOutputqueue0/40,535drops;inputqueue0/75,1716drops5minuteinputrate0bits/sec,0packets/sec5minuteoutputrate0bits/sec,0packets/sec

Show命令(Con‘t)15628691packetsinput,2423133689bytes,0nobufferReceived698983broadcasts,0runts,0giants0inputerrors,0CRC,0frame,0overrun,0ignored,0abort0watchdog,0multicast0inputpacketswithdribbleconditiondetected25793600packetsoutput,4210267836bytes,0underruns0outputerrors,0collisions,3interfaceresets0babbles,0latecollision,0deferred0lostcarrier,0nocarrier0outputbufferscopied,0interrupts,0failuresShow命令rtr1-c-1-bjbj>shints0/0/0Serial0/0/0isup,lineprotocolisupHardwareiscyBusSerialDescription:<bjbj-mci-c-2M-01>toMCI(BEIJING-SACRAM2048K003)MTU1500bytes,BW1984Kbit,DLY20000usec,rely255/255,load1/255EncapsulationHDLC,loopbacknotset,keepaliveset(10sec)Lastinput00:00:07,output00:00:07,outputhangneverLastclearingof"showinterface"counters18w4dQueueingstrategy:fifoOutputqueue0/40,455018414drops;inputqueue0/75,124drops5minuteinputrate0bits/sec,0packets/sec5minuteoutputrate15000bits/sec,1packets/sec110103206packetsinput,3264989278bytes,1nobufferReceived0broadcasts,0runts,0giants11447inputerrors,9191CRC,0frame,28overrun,0ignored,2228abort3720562109packetsoutput,503074661bytes,0underruns0outputerrors,0collisions,534interfaceresets0outputbufferscopied,0interrupts,0failures214carriertransitions6921alarmindications,6959remotealarms,7003rxLOF,42rxLOSDCDup,BERinactive,NELRinactive,FELRinactiveShow命令rtr1-c-1-bjbj>shproccpuCPUutilizationforfiveseconds:6%/5%;oneminute:11%;fiveminutes:9%PIDRuntime(ms)InvokeduSecs5Sec1Min5MinTTYProcess12627167202873954159140.57%0.45%0.47%0BGPRouter23058357443100720986350.00%0.91%1.16%0Checkheaps3547204088011330.00%0.00%0.00%0PoolManager40200.00%0.00%0.00%0Timers50100.00%0.00%0.00%0OIRHandler60100.00%0.00%0.00%0IPCZoneManager70100.00%0.00%0.00%0IPCRealmManager85196220447720.00%0.00%0.00%0IPCSeatManager9367268719850.00%0.00%0.00%0ARPInput100100.00%0.00%0.00%0SERIALA'detect110100.00%0.00%0.00%0MicrocodeLoader120100.00%0.00%0.00%0ATMILMIInputShow命令rtr1-c-1-bjbj>shversionCiscoInternetworkOperatingSystemSoftwareIOS(tm)GSSoftware(RSP-JV-M),Version11.1(19)CC1rtr1-c-1-bjbjuptimeis1year,14weeks,2days,16hours,0minutesSystemrestartedbyreloadat01:23:52PRCSunJan311999Systemimagefileis"slot0:rsp-jv-mz.111-19.CC1",bootedviaslot0ciscoRSP4(R5000)processorwith262144K/2072Kbytesofmemory.R5000processor,Implementation35,Revision2.1(512KBLevel2Cache)Lastresetfrompower-onChassisInterface.5VIP2R5Kcontrollers(2FastEthernet)(9Serial)(2HSSI)(2POS).2FastEthernet/IEEE802.3interfaces.9Serialnetworkinterfaces.2HSSInetworkinterfaces.2PacketoverSonetnetworkinterfaces.Configurationregisteris0x102Debug命令r4#debugarpARPpacketdebuggingisonr4#0:33:08:IPARPreqfilteredsrc0800.209f.92b9,dst2ffff.ffff.ffffmartiansource0:33:09:IPARPreqfilteredsrc0800.209f.92b9,dst0ffff.ffff.ffffmartiansource0:33:09:IPARPreqfilteredsrc0800.209f.92b9,dst0ffff.ffff.ffffmartiansource0:33:09:IPARPreqfilteredsrc0800.209f.92b9,dst2ffff.ffff.ffffmartiansource0:33:09:IPARP:rcvdreqsrc680000.21d9.5722,dst6Ethernet00:33:09:IPARPreqfilteredsrc0800.209f.92b9,dst0ffff.ffff.ffffmartiansource0:33:09:IPARPreqfilteredsrc0800.209f.92b9,dst3ffff.ffff.ffffmartiansource0:33:09:IPARPreqfilteredsrc0800.209f.92b9,dstffff.ffff.ffffmartiansourceDebug命令r4#debugippacketIPpacketdebuggingisonr4#0:36:49:IP:s=1(Ethernet0),d=55,len118,rcvd00:36:49:IP:s=4(Ethernet0),d=55(Ethernet0),len234,rcvd30:36:49:IP:s=89(Ethernet0),d=55,len140,rcvd00:36:49:IP:s=89(Ethernet0),d=55,len140,rcvd00:36:49:IP:s=36(Ethernet0),d=55,len138,rcvd00:36:49:IP:s=28(Ethernet0),d=55,len140,rcvd00:36:49:IP:s=1(Ethernet0),d=55,len118,rcvd00:36:49:IP:s=34(Ethernet0),d=55,len146,rcvd00:36:49:IP:s=0(Ethernet0),d=55(Ethernet0),len231,rcvd30:36:49:IP:s=(Ethernet0),d=55,len146,rcvd00:36:49:IP:s=(Ethernet0),d=55(Ethernet0),len231,redirected0:36:49:IP:s=(Ethernet0),d=55(Ethernet0),g=55,len231,forwardLogging命令Loggingon四种方式console-loggingconsolelevelbuffer-loggingbufferedunixsyslogloggingip-addressloggingtraplevelmonitor-loggingmonitorlevelUNIXSyslog主机Syslog

configfile/etc/syslog.confForexample:local7.warn/var/log/local7.warn(makesure“tab”betweenlocal7.warnand/var/log/local7.warn,notspace)Thefilelocal7.warnmustexistandbewritable

syslogdmustberestartedbyroot

ps-ef|grep

syslogd kill-9<pid>

syslogd-dErrormessageloggingConsole(资源消耗最大)TerminalmonitorsyslogdbufferSyslog优先级Syslog文件sendmail[1138]:NOQUEUE:4:fl=0x1,mode=20666:CHR:size=0sendmail[1138]:PAA01138:from=wangjk,size=160,class=0,pri=30160,nrcpts=1,msgid=<200001090717.PAA01138@tech.asiainfo>,relay=wangjk@localhostsendmail[1140]:PAA01138:to=root,ctladdr=wangjk(237/200),delay=00:00:01,xdelay=00:00:01,mailer=local,stat=Sentsendmail[225]:startingdaemon(8.9.3):SMTP+queueing@00:15:00sendmail[457]:NOQUEUE:Nullconnectionfromlocalhost[]sendmail[216]:startingdaemon(8.9.3):SMTP+queueing@00:15:00Solaris系统命令Ping命令Ping命令结果unknownhost--不能解析远程主机的域名networkunreachable--本地主机没有到远程主机的相应的路由noanswer--没有得到远程主机的响应hostisalive--得到远程主机的响应tracerouteTime-to-live值由1开始递增值为1非法的UDP端口默认值为33343tracerouteSource1stGateway2ndGatewayDestinationttl=1decreaseto0returnerrordecreaseto0returnerrordecreaseto1forwarddecreaseto1forwarddecreaseto2forwardreachdestinationportunreachablereturnerrorttl=2ttl=3Ifconfig命令/disk1/shao%ifconfig-alo0:flags=849<UP,LOOPBACK,RUNNING,MULTICAST>mtu8232inetnetmaskff000000hme0:flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST>mtu1500inetnetmaskNslookup命令/disk1/shao%nslookupNetstat命令/disk3/shao%netstat-rnRoutingTable:DestinationGatewayFlagsRefUseInterface-----------------------------------------------------------------4UGHD031U3593le031U30le0defaultUG037UH00lo0Netstat命令/disk3/shao%netstat-iNameMtuNet/DestAddressIpktsIerrsOpktsOerrsCollisQueuelo08232loopbacklocalhost42042000le01500tech.asiainfotech81343104139631250Arp命令/disk3/shao%arp-aNettoMediaTableDeviceIPAddressMaskFlagsPhysAddr-------------------------------------------------------------le0proxy5508:00:20:7a:fb:82le05500:a0:c9:8a:a8:f9le0ai-nas5500:b0:64:e9:d8:41le05508:00:20:b0:85:18le0aints15500:a0:c9:8d:d9:95le065500:d0:59:08:28:50le005500:00:21:00:92:98le0aints35500:90:27:c3:bb:d5le05508:00:20:9f:92:b9厂家MAC地址列表一般性故障诊断RouterCrashWhatiscrash?SystemhasdetectedanunrecoverederrorandrestarteditselfWhattocollect?ShowtechShowlogShowenvironmentallDonotreloadrouterbeforegettingallnecessaryinformationRouterHangWhatisrouterhang?Systemencounteredacatastrophicfailurebutcan’trecoveritselfWhattodonext?CheckiftrafficcanstillflowthroughrouterIfyes,disconnectallinterfacestoseeifconsoleisavailable;Ifconsoleisavailable,doshowtech,showlog,showenvironall;ifnot,pressbreakkeytorommonmodeRouterHang(cont.)Ifnotrafficcangetthrough,presscrtl-breakkeytorommonmodeInrommonmode,typethefollowingcommandsandgetalltheoutputK,context,contandwaitingforawhileDothesame:K,context,contRepeataboveforseveraltimes（Breakmustbeenabledinconfig-registerforbreakingtorommonconfig-register0x2002reloadtotakeeffect）MemoryProblemsMemoryfragmentation

memorymaygetfragmentedintoverysmallblockssothatallprocessesthatrequirealargerblockcannotgetmemoryoftheblocksizethatitneeds.Memoryleakmemoryleakmeansthataparticular'process'isnotfreeingupanyallocatedmemorywhenthisprocessisdonewithit.ThisprocesskeepstheRAMinuseevenwhenitdoesn'tneedit.

LackofMemory局域网故障诊断有关的路由器命令ping及trace命令showarpshowinterfaceshowiprouteshowipaccess-listdebug局域网故障诊断有关Solaris系统命令arp命令ping/traceroute命令ifconfig命令netstat命令nslookup命令局域网故障诊断流程本地主机配置正确?能否ping本地路由器?所有路由器是否可达?远端主机配置正确?NYYYNNN修改本地主机配置(address,mask,defaultgateway)修改本地路由器配置及局域网问题修改所有路由器配置及广域网问题修改远端主机配置现象与故障判断现象与故障判断(Con’t)广域网故障诊断广域网故障诊断相关的路由器命令showinterfaceserialshowlineshowcontrollerping/traceroutedebug路由协议故障诊断一般性命令showipprotocolshowiproutesunmmaryOSPF路由协议showipospfdatabaseshowipospfinterfaceshowipospfneighborBGP路由协议showipbgpshowipbgpsummaryshowipbgpcommunityIS-IS路由协议showclnsneighborshowclnsrouteshowclnsis-neighborshowisisdatabase串行链路环测Debug命令Debug命令debugserialinterfacedebugpppnegotiationdebugserialpacket慎重地使用debug命令!Debug命令路由协议诊断rtr1-c-1-bjbj>shiproutesumRouteSourceNetworksSubnetsOverheadMemory(bytes)connected086321088static01155169772157080isis44152872856984Level1:0Level2:419bgp649997275712937479886411842064External:1463Internal:84113Local:118internal2121288456Total7488214515499799612345672路由协议诊断-OSPFr1-c-bj>shipospfdatabase

OSPFRouterwithID(2)(ProcessID165)

RouterLinkStates(Area0)

LinkID

ADVRouter

Age

Seq#

841

0x800032DB0x38E5

1421

0x800017AC0x7AFF

384

0x8000155F0x6F25

1725

0x8000066A0xB478

1558

0x800022160xCD0E

1092

0x80000CBB0xAFB3

1499

0x800049C90x7D75

850

0x800010A80xDE95

1450

0x800037760x228E

764

0x800002150x43BD

463

0x800031F60x1E32

21

路由协议诊断-OSPFr1-c-bj>shipospfint

FastEthernet0/1/0isup,lineprotocolisup

InternetAddress/29,Area0

ProcessID165,RouterID2,NetworkTypeBROADCAST,Cost:1

TransmitDelayis1sec,StateDR,Priority1

DesignatedRouter(ID)2,Interfaceaddress

BackupDesignatedrouter(ID)1,Interfaceaddress

Timerintervalsconfigured,Hello10,Dead40,Wait40,Retransmit5

Helloduein00:00:09

Index2/2,floodqueuelength0

Next0x0(0)/0x0(0)

Lastfloodscanlengthis3,maximumis40

Lastfloodscantimeis0msec,maximumis4msec

NeighborCountis1,Adjacentneighborcountis1

(BackupDesignatedRouter)

Suppresshellofor0neighbor(s)

路由协议诊断-OSPFr1-c-bj>shipospf

neighbor

NeighborID

Pri

State

DeadTime

Address

1

FULL/BDR

00:00:35

1

FULL/

-

00:00:35

1

FULL/

-

00:00:37

1

FULL/

-

00:00:36

1

FULL/

-

00:00:37

1

FULL/

-

00:00:36

1

FULL/

-

00:00:38

1

FULL/

-

00:00:34

1

FULL/

-

00:00:34

1

FULL/

-

00:00:36

1

FULL/

-

00:00:34

1

FULL/

-

00:00:34

ATM4/0/0.11

路由协议诊断-BGPrtr1-c-1-bjbj>shipbgpStatuscodes:ssuppressed,ddamped,hhistory,*valid,>best,i-internalOrigincodes:i-IGP,e-EGP,?-incompleteNetworkNextHopMetricLocPrfWeightPath*500699370180i*45500356170180i*i10001492370180i*>i10001492370180i*50069931i*204.189.216*>i69100057271i路由协议诊断命令-BGPrtr1-c-1-bjbj>shipbgpsumBGPtableversionis41225605,mainroutingtableversion4122560585846networkentries(500146/591202paths)using27630064bytesofmemory64443BGPpathattributeentriesusing8667060bytesofmemory31859BGProute-mapcacheentriesusing509744bytesofmemory19590BGPfilter-listcacheentriesusing313440bytesofmemoryDampeningenabled.120historypaths,376dampenedpaths165904receivedpathsforinboundsoftreconfiguration32583269prefixesrevised.NeighborV