常见网络故障诊断解析课件

设备维护和故障诊断

设备维护和故障诊断路由器设备维护和故障诊断

路由器设备

ShowInterface ShowInterface缓存和队列(什么是Ignore?)当没有缓存可以被用来储存输入的帧时，Ignore就会增加队列极限值可以被超过以便处理突发的流量InterfaceBusInterface

CardInterface

Card123x123yEthernetøTokenRing1

PacketBuffersGlobalPoolSystemBuffersCPUT1T2TyTy+8Ex+2Ex+1ExE21,2,3…x1,2,3…x••••••••缓存和队列(什么是Ignore?)当没有缓存可以被用来储存缓存和队列(InputDrops)当没有足够的系统缓存可用时Inputdrop值就会增加系统缓存被用作处理所有经处理交换和路由器自身生成的数据包InterfaceBus

InterfaceBuffersSystemBuffersCPUNOVACANCYXEthernetFrame1•

•

••

•

••

•

•Ethernetø1,2,3…x缓存和队列(InputDrops)当没有足够的系统缓存可缓存和队列(OutputDrops)当没有足够的接口缓存可用作处理输出帧时，Outputdrop值便会增加InterfaceBus

InterfaceBuffersSystemBuffersCPUTokenRingXNewFrameNOVACANCY缓存和队列(OutputDrops)当没有足够的接口缓存可缓存和队列(OutputDrops)当超出输出接口的队列极限值时，Outputdrop值会增加InterfaceBusSTOP

InterfaceBuffersSystemBuffersCPUNewFrame••

•

•

•TokenRing1缓存和队列(OutputDrops)当超出输出接口的队列极接口缓存HoStage>showcontrollercbuscBus0,controllertype6.0,microcodeversion10.0

512Kbytesofmainmemory,128Kbytescachememory1341520bytebuffers,654496bytebuffers

Restarts:0linedown,0hungoutput,0controllererrorMEC0,controllertype5.1,microcodeversion10.0

Interface0-Ethernet0,stationaddress0000.0c06.4ae0(bia0000.0c06.4ae0)11bufferRXqueuethreshold,18bufferTXqueue

limit,buffersize1520

ift0000,rql11,

tq00000000,tql18

Transmitterdelayis0microsecondsCTR1,controllertype9.0,microcodeversion10.1

Interface8-TokenRing0,stationaddress0000.3060.3219(bia0000.3060.3219)13bufferRXqueue

threshold,31bufferTXqueue

limit,buffersize4496

ift0005,rql13,tq00000000,tql31

Transmitterdelayis0microsecondsFDDI-T3,controllertype7.2,microcodeversion10.1

Interface24-Fddi0,stationaddress0000.0c06.36d7(bia0000.0c06.36d7)13bufferRXqueue

threshold,32bufferTXqueue

limit,buffersize4496

ift0006,

rql9,tq00000000,tql32接口缓存HoStage>showcontrollerc系统缓存HoStage#showbuffersBufferelements:

500infreelist(500maxallowed)

51640224hits,0misses,0createdSmallbuffers,104bytes(total121,permanent121):

119infreelist(20min,250maxallowed)

19229201hits,0misses,0trims,0createdMiddlebuffers,600bytes(total90,permanent90):

89infreelist(10min,200maxallowed)

20513359hits,91misses,115trims,115createdBigbuffers,1524bytes(total90,permanent90):

90infreelist(5min,300maxallowed)

7160285hits,0misses,0trims,0createdLargebuffers,5024bytes(total5,permanent5):

5infreelist(0min,30maxallowed)

233295hits,0misses,0trims,0createdHugebuffers,18024bytes(total0,permanent0):

0infreelist(0min,4maxallowed)

0hits,0misses,0trims,0created系统缓存HoStage#showbuffersBuff命令ShowprocesscpuCPUutilizationforfiveseconds:X%/Y%;oneminute:Z%;fiveminutes:W%PIDRuntime(ms)InvokeduSecs5Sec1Min5MinTTYProcess下面的表格解释了命令输出的各详细参数意思:

FieldDescriptionXAveragetotalutilizationduringlastfivesecondsYAverageutilizationduetointerrupts,duringlastfivesecondsX-YThisrepresentspercentageoftrafficbeingprocessswitchedZAveragetotalutilizationduringlastminute*WAveragetotalutilizationduringlastfiveminutes*PIDProcessIDRuntimeCPUtimetheprocesshasused(inmilliseconds)InvokedNumberoftimesaprocesshasbeencalleduSecsMicrosecondsofCPUtimeforeachinvocation5SecCPUutilizationbytaskinthelast5seconds1MinCPUutilizationbytaskinthelastminute*5MinCPUutilizationbytaskinthelast5minutes*TTYTerminalthatcontrolstheprocessProcessNameofprocess命令ShowprocesscpuCPUutiliza高利用率分析router-5#showprocesscpu

CPUutilizationforfiveseconds:83%/21%;oneminute:79%;fiveminutes:84%

PIDRuntime(ms)InvokeduSecs5Sec1Min5MinTTYProcess11043707280.00%0.00%0.00%0LoadMeter210208152226700.00%0.01%0.00%0OSPFHello334620579597920.00%0.20%0.17%0Checkheaps总共83%的利用率，其中21%是中断83-21==62%实时运行的进程高利用率分析router-5#showprocesscp

中断引起的高CPU利用率CPU中断的主要原因是数据流量的快速交换路由器上配置了语音端口

路由器上有活跃的异步传输(ATM)端口路由器上配置了不正确的交换路径CPU在处理内存修正路由器高负荷运转检查端口show命令的输出IOS软件有bug

中断引起的高CPU利用率CPU中断的主

进程引起的高CPU利用率

如果一个进程占用了大量的CPU资源，检查log信息。进程上的不寻常活动会引起log中的错误信息。下面这些进程会引起CPU的高利用率:

IPInput

HyBridgeInput

IPSimpleNetworkManagementProtocol(SNMP)

VirtualEXEC

TCPTimer

VTEMPLATEBackgr

OtherProcesses

进程引起的高CPU利用率如果一个进程占用了大量ShowmemoryHeadTotal(b)Used(b)Free(b)Lowest(b)Largest(b)Processor815D3828154402326758292868194028271443901312I/O2400000125829121625120109577921085858010871708

ProcessormemoryAddressBytesPrevNextRefPrevFNextFAllocPCwhat815D86740000001500815D3828815D8C7C001----------------801E14DCListElements815D8C7C0000005000815D8674815DA030001----------------801E1518ListHeaders815DA0300000000044815D8C7C815DA088001----------------80D6FCCC*Init*815DA0880000001500815DA030815DA690001----------------801EBD04messages主内存包交换内存每个分配的内存块Showmemory主内存包交换内存每个分配的内存块内存问题

MALLOCFailures（内存分配失败）MemoryLeaks（内存漏洞）Fragmentation（碎片）AlignmentErrors（修正错误）SpuriousAccesses（虚假存取）MemoryCorruption（内存崩溃）ProcessorMemoryParityErrors（处理器内存校验错误）内存问题MALLOCFailures（内存分配失败）Showprocessmemory

Router>showprocessesmemoryTotal:3149760,Used:2334300,Free:815460PIDTTYAllocatedFreedHoldingGetbufsRetbufsProcess002265481252180437600*Initialization*00320542228832000*Scheduler*0056636922173356018561000*Dead*10264264378400LoadMeter22570053721312400VirtualExec3000678400Checkheaps40960688000PoolManagerAllocated=路由器启动后分配给进程的总字节数Freed=进程释放的总字节数Holding=进程拥有的总字节数。这是进程拥有的实际字节数，是故障诊断中最重要的判断依据。这个数值不一定等于Allocated减去Freed，因为有些进程在分配了一块内存后，会被另外的进程返回到空闲池中。ShowprocessmemoryRouter>sho交换机设备维护和故障诊断

交换机设备自动协商总结Peer1配置Peer2配置Peer1结果Peer2结果注释AutoAuto100FD100FDCorrectnegowhenbothpeerarecapableof100FD100FDAuto100FD100HDDUPLEXMISMATCH100FD100FD100FD100FDCorrectmanualconfig100HDAuto100HD100HDLinkisestablished,butpeer2doesnotseeanyauto-negotiationinformationfromNICanddefaultstohalf-duplex.10HDAuto10HD10HDLinkisestablished,butpeer2willnotseeFLPandwilldefaultto10Mbpshalf-duplex.10FD100FDNolinkNolinkSPEEDMISMATCH自动协商总结Peer1配置Peer2配置Peer1结果P检查双工不匹配CDP将会在第一次连接时告警全双工意味着碰撞监测机制不启用，一个全双工的设备将会不检查传输介质是否空闲就直接发送数据帧双工不匹配的症状FCSerrors(seenonFDside)Alignerrors(seenonFDside)Runts(seenonFDside)Excessivecollision(seenonHDside)Latecollision(seenonHDside)检查双工不匹配CDP将会在第一次连接时告警总结:10/100M自动协商 尽量使用:Auto对auto固定的speed/duplex对固定的speed/duplex避免:Auto对固定的speed/duplex总结:10/100M自动协商 尽量使用:千兆以太协商问题有些设备不支持千兆协商或部分支持如果问题引起linkup，disable千兆连接的协商协商需要同时在两端enabled或者disabled千兆以太协商问题有些设备不支持千兆协商或部分支持千兆以太协商问题TwoswitchesAandBareconnectedViaGigEther千兆以太协商问题TwoswitchesAandBaCatOS如何监控Shport:显示端口状态和一些错误计数器（counters）Shmac:显示端口下Rx和TX的流量数据Shcountersmod/port:显示所有的计数器Shtop:在30秒内数据流量最大的10个端口

快速诊断广播风暴的源和环路端口的方法CatOS如何监控Shport:显示端口状态和一些XL交换机如何监控Shinterface:显示端口信息，类似于IOS里的shint命令Shcontrollerethernet[fast|gig]x/x:显示更多的端口计数器数值XL交换机如何监控Shinterface:显示端口常见的端口问题坏的连接线(失效,错误的类型):导致连接无法建立,FCS,runt,align,….(GigEModeconditioningcablerequiredforLX/LH-GBICwithMMCabledist<100mor>300m)(ZX-GBICisforextendeddistances,minimumdistancewith8dbattenuator10kmmin,withoutattenuator40kmmin)双工不匹配:通常CDP协议会报告双工不匹配，但不会自动修复CatOS下的端口状态未激活(状态LED灯为桔黄色):通常由于端口分配到一个不存在的

(VTP的问题见后)常见的端口问题坏的连接线(失效,错误的类型):导致连接常见的端口问题:err-disable端口状态为err-disable(仅对于CatOS)的原因是:某个端口上的大量错误EtherChannel配置错误BDPU端口告警其他r…如果端口进入err-disable状态:设置端口选项errportenable这步操作防止所有端口进入err-disable状态，在需要时使用最好先找到err原因!!常见的端口问题:err-disable端口状态为err-常见的端口问题:err-disableerr-disable的端口需要手工的re-enabled可以设置err-disable的端口在x秒后重新enable:Taras>(enable)seterrdisable-timeoutUsage:seterrdisable-timeout<enable|disable><reason>seterrdisable-timeoutinterval<interval>(reason=bpdu-guard,channel-misconfig,duplex-mismatch,udld,other,allinterval=30..86400seconds)常见的端口问题:err-disableerr-disabl五种Trunking模式off

表示Trunk不会建立auto(默认配置)表示会响应Trunk协商但不会主动发起协商(它不会主动trunk,但如果对端要求Trunk，也会参与协商)desirable

表示连接将会协商并主动建立Trunk五种Trunking模式off表示Trunk不会建立Trunkingon

表示建立Trunk并发送DTP数据包(需要设置封装方式ISL或者dot1q)Nonegotiate表示建立Trunk但不发送DTP数据包nonegotiate应该在Trunk不稳定(trunk/non-trunk)的情况下暂时使用，或者是设备不支持DTP协议比如路由器或者XL系列的交换机。Trunkingon表示建立Trunk并发送DTP数据包Trunking建议在核心和边界分别配置desirable

对

auto

，或者是desirable

对desirable如果连接必须是Trunk，配置on

对onTrunking建议在核心和边界分别配置desirableTrunking总结Trunk或连接上的问题:确认端口连接在innontrunk模式确认至少一端是desirable模式，或者两端都是on模式，然后检查两端的封装方式是否一致做一次showmac来检查in-discards数值不再增长检查VTP域名为TAC捕捉以下信息：Shtrunk(orshintx/xswitchport)Shspantx/x(orshspanningintx/x)ShconfigTrunking总结Trunk或连接上的问题:FEC/PAgPFEC也有on,off,auto,anddesirable

几种配置状态，但含义上仅有微小的区别on

表示端口会进入channel但不会运行PAgP(PortAggregationProtocol)FEC/PAgPFEC也有on,off,autFECauto

表示端口会响应协商(他们监听PAgP)但不会主动形成channeldesirable

表示端口会收发PAgP并主动建立channelFECauto表示端口会响应协商(他们监听PAgP)FEC警告:autotoon不会工作警告:ontoon会工作但不会运行PAgP推荐desirabletodesirable

，如果两端都支持PAgPon

用于连接一个不支持PAgP的设备(连接路由器或者一个XL系列的交换机)FEC警告:autotoon不会工作FEC如果错误的配置了FEC，或者FEC监测到SpanningTree环路,它会将端口设成ERR-DISABLE状态同一个channel中的所有端口都必须有相同的配置参数:同样的speed,duplex,trunkingstatus,DTPconfig,vlanallowed,….FEC如果错误的配置了FEC，或者FEC监测到Spann以太网Channel总结和要点Shportcapa

告诉channel容量检查两端的工作模式(比如，最好是desirabletodesirable或者ontoon的配置,不要autotoauto或其他混和方式)如果channel起不来，尝试以下处理方法停掉所有端口的trunking确认speed,duplex,nativevlan匹配，并放入channel一旦channel起来,在第一个端口配置trunk，然后trunk的配置会被拷贝至其余的所有端口以太网Channel总结和要点ShportcapaSpanningTreeTroubleshootingSpanningTreeTroubleshootingWhatCausesLoops?

1)ConfigurationproblemsSpantreedisabledSpantreeenabledonsomeswitchesbutnotonothersSpeed/duplexmismatchesPortfastenabledonportsconnectedtohubsorswitchesRouter,multiportNIC,configuredforbridgingUsingdifferentspantreeprotocolswithinthesameVLANMisconfiguredorbuggytrunk-orchannel-capableNICLoopswithhubsorswitchesPortchannelingmisconfigurationWhatCausesLoops?1)ConfigurWhatCausesLoops?

2)DesignissuesToolargeofaswitchednetworkBridgingovertheWAN(delayproblems)WhatCausesLoops?2)DesigniWhatCausesLoops?

3)SoftwareissuesSoftwarebugsForwardingtrafficacrossblockedportsUplinkFast/BackboneFastEtc.LossofmanagementcommunicationtolinecardsWhatCausesLoops?3)SoftwarWhatCausesLoops?

4)HardwareIssuesLayeronelinksthatarebad(i.e.CRCs,otherinputerrors)UnidirectionallinksDatacorruption(BPDUsdropped)PortStuck(BPDUsdropped)NMPstopslisteningtospanning-tree(stuckinband)LossofmanagementcommunicationtolinecardsWhatCausesLoops?4)HardwareDetectingSpanningTreeLoops1)NetworkisEXTREMELYslowforallnodes2)Networkoutage3)Highsystemutilizationonswitch

SystemUtilizationin“showsystem”above20%usuallyindicatesaloopAbove7%indicatespossibletransitoryloopDependsonnetworktrafficandhardware(Cat5000Sup1vs.Cat6000Sup2,etc.)4)SystemLEDindicatorsonSwitchUtilizationBar5)HighAmountofIn-lostandOut-loston“showmac”6)HSRP,OSPF,etcreportduplicateIPaddress7)UnicastfloodingDetectingSpanningTreeLoops1DetectingSpanningTreeLoopsCheckspantreeblockedandrootportsforerrorsusing“showport”,“showmac”&“showcounters”Setupasyslogserverandturnonloggingforthe“spantree”facilityto6,whichwillshowporttransitionsthroughthespantreestates(listening,learning,etc.)Use“showinband”tocheckfor“RsrcErrors”(BPDUcouldbedroppedifsupervisorisunabletoprocesstheBPDU)Checktoseeifyouareexceedingspanningtreeinstances“showspantreesummary”DetectingSpanningTreeLoopsCDuringanEventRemoveredundantEthernetsegmentsfromthenetworkStartwithconnectionsbetweencoreswitchesBeginwithEtherChannels,ifusedWaitfor30-60secondsforthenetworktorecoverbeforeremovinganotherlinkIfthenetworkdoesnotrecover,continuemethodicallyremovingredundancyuntilthenetworkstabilizesAvoidrebootingorpoweringoffswitchesIfyoudothisyou’lllosetheloggingbuffer&spantreestatsontheswitchSyslogtoaservercannotnecessarilybetrustedduringanetworkfailureDuringanEventRemoveredundan

FindingtheSmokingGunUse“showsystem”tofindswitcheswithhighbackplaneutilizationUse“showmac”andlookforlargeamountsofbroadcast/multicastreceived&transmittedUse“showspantreestatistics”tofollowtheproblemthroughthenetworkOntheroot,checkthe“topologychangeinitiator”toseewhichbridgelastgeneratedaTCNLookfor“msgageexpirycount”onblockedportstoseewhetherweexpiredaBPDUontheport(MaxAgewasreached)Lookfor“tcnbpdu'sxmitted”toseewhetherabridgesentmanyTCNsLookfor“forwardtranscount”toseehowmanytimestheporttransitionedintotheforwardingstateFindingtheSmokingGunUse“s

PreparingfortheNextTimeTakeproactivemeasures(performthesetaskspriortohavinganotherevent)Turnspantreelogginglevelontheswitchesto6(“setlogginglevelspantree6default”)toseestatetransitions&TCNs(also,logtoaserver)OnswitchesrunningIOS,use“debugspanningevents”Enter“clearcounters”onallswitchesPreparingfortheNextTimeTaFindingtheRootVerifythelocationoftherootThecustomermighthavefailedtodeterministicallysettherootTherootmighthavemovedduetoanewbridgeinthenetwork,orabridgeprioritychangeesc-cat6500-a>(enable)showspantree5VLAN5SpanningtreeenabledSpanningtreetypeieeeDesignatedRoot00-d0-06-26-f4-04DesignatedRootPriority8192DesignatedRootCost3DesignatedRootPort2/1-2(agPort13/33)RootMaxAge20secHelloTime2secForwardDelay15secBridgeIDMACADDR00-d0-bb-01-30-04BridgeIDPriority32768BridgeMaxAge20secHelloTime2secForwardDelay15secPortVlanPort-StateCostPriorityPortfastChannel_id--------------------------------------------------------------------------2/1-25forwarding332disabled80115/15forwarding432enabled0ThebridgeIDoftherootbridgeRootport(porttogettorootbridge)FindingtheRootVerifythelocesc-6500-b>(enable)showspantree5VLAN5SpanningtreeenabledSpanningtreetypeieeeDesignatedRoot00-d0-06-26-f4-04DesignatedRootPriority8192DesignatedRootCost0DesignatedRootPort1/0RootMaxAge20secHelloTime2secForwardDelay15secBridgeIDMACADDR00-d0-06-26-f4-04BridgeIDPriority8192BridgeMaxAge20secHelloTime2secForwardDelay15secPortVlanPort-StateCostPriorityPortfastChannel_id--------------------------------------------------------------------------4/1-25forwarding332disabled865esc-6500-b>(enable)showspantreesummaryRootswitchforvlans:4-10.

FindingtheRootRootIDandBIDwillmatchontherootbridgeDesignatedrootcostontherootisalways“0”In5.4andlater,use“showspantreesummary”toseeforwhichVLANstheswitchisrootesc-6500-b>(enable)showspanesc-6500-b>(enable)showspantreesummary<...>SummaryofconnectedspanningtreeportsbyvlanVlanBlockingListeningLearningForwardingSTPActive--------------------------------------------------1200464000225000666000447000448000449000441000044

BlockingListeningLearningForwardingSTPActive--------------------------------------------------Total2003234<...>FindingActiveandBlockedPortsTotalblockingportsontheswitchTotalportsinthespanningtree(donotexceedlimitsspecifiedforyoursupervisorengineintheReleaseNotesesc-6500-b>(enable)showspanViewingBlockedPortsesc-6500-b>(enable)showspantreeblockedT=trunkg=groupPortsVlans---------------8/23(T)18/24(T)1Numberofblockedports(segments)inthesystem:2Ports8/23and8/24areblockingforVLAN1ViewingBlockedPortsesc-6500-MonitoringBlocked&RootPortsesc-6500-b>(enable)showspantreestat8/231Port8/23VLAN1SpanningTreeenabledforvlanNo=1BPDU-relatedparametersportspanningtreeenabledstateblockingport_id0x836cportnumber0x36cpathcost12messageage(port/VLAN)3(20)designated_root00-30-94-93-e5-80designated_cost19designated_bridge00-50-53-59-a0-00designated_port0x8001top_change_ackFALSEconfig_pendingFALSEport_inconsistencynonePORTbasedinformation&statisticsconfigbpdu'sxmitted(port/VLAN)36(698871)configbpdu'sreceived(port/VLAN)215843(608891)tcnbpdu'sxmitted(port/VLAN)0(7)Blocked&rootportsshouldreceiveBPDUsevery2secondsMonitorblockedandrootportstoseeiftheyarereceivingconfigBPDUsevery2secondsCheckforerrorsonblockedorrootports,whichmightcauseablockedporttotransitionoutofblockingmode,orarootbridgechangePorts8/23isblockingforVLAN1Makesurethe“configbpdu’sreceived”counterisincrementingontheportapproximatelyevery2secondsMonitoringBlocked&RootPortIfBPDUsarenotbeingreceivedevery2seconds(oratall)ontheport,checkforerrorsusing:showportcounters–CheckforLayer1errors(Align,FCS,etc.)showmac–Makesurethe“Rcv-Multicast”counterisincrementing;makesurethe“In-Discard”counterisnotincrementingshowcounters–Checkforanyerrorsonthereceivesideshowinband–Lookfor“RsrcErrors”showcamsystem–Makesure01-80-c2-00-00-00(IEEE802.1dBPDUMAC)islistedasasystementryfortheVLANMonitoringBlocked&RootPortsIfBPDUsarenotbeingreceiveMonitoringSpanningTreeConsole>(enable)showspantree3/47PortVlanPort-StateCostPriorityPortfastChannel_id--------------------------------------------------------------------------3/471blocking301932disabled03/472blocking301932disabled03/473blocking301932disabled03/474forwarding301932disabled03/475forwarding301932disabled03/476forwarding301932disabled03/4710forwarding301932disabled03/4711forwarding301932disabled0Console>(enable)showspantree3activeVLAN3SpanningtreeenabledSpanningtreetypeieeeDesignatedRoot00-50-80-39-ee-42DesignatedRootPriority32768DesignatedRootCost3019DesignatedRootPort3/48RootMaxAge20secHelloTime2secForwardDelay15secBridgeIDMACADDR00-d0-00-3f-a0-02BridgeIDPriority49152BridgeMaxAge20secHelloTime2secForwardDelay15secPortVlanPort-StateCostPriorityPortfastChannel_id--------------------------------------------------------------------------3/473blocking301932disabled03/483forwarding301932disabled04/1-43forwarding300232disabled865MonitoringSpanningTreeConsolConsole>(enable)showspantreestatistics3/473

Port3/47VLAN3SpanningTreeenabledforvlanNo=3BPDU-relatedparametersportspanningtreeenabledstateblockingport_id0x80afportnumber0xafpathcost3019messageage(port/VLAN)0(20)designated_root00-50-80-39-ee-42designated_cost0designated_bridge00-50-80-39-ee-42designated_port0x8026top_change_ackFALSEconfig_pendingFALSEport_inconsistencynonePORTbasedinformation&statisticsconfigbpdu'sxmitted(port/VLAN)2(127624)configbpdu'sreceived(port/VLAN)51(3124)tcnbpdu'sxmitted(port/VLAN)0(65)tcnbpdu'sreceived(port/VLAN)0(36)forwardtranscount0scpfailurecount0Spanning-treeportstateConfigBPDUstatsforport&VLANTCNBPDUstatsforport&VLANNumberoftimestheporttransitionedtoforwardingmodeMonitoringSpanningTreeConsole>(enable)showspantre[continued]VLANbasedinformation&statisticsspanningtreetypeieeespanningtreemulticastaddress01-80-c2-00-00-00bridgepriority49152bridgemacaddress00-d0-00-3f-a0-02bridgehellotime2secbridgeforwarddelay