《业务故障排错方法》PPT课件.ppt

9. 业务故障排错方法,各业务trouble shooting方法介绍,1、pppoe业务的trouble shooting 比较常见的报错是678、718、691错误，其中678最复杂 。 1.1、先说678,E320,8512,DSLAM,PC,以上图为列，678之所以复杂，是因为在传输的各环节中任何一点出现问题，都可能造成678.,检查思路 先检查bas，通过bas上的检查来确定下一步检查步骤 对链路进行检查时，必须在汇聚交换机上进行镜像抓包和隔离传输设备进行测试的方式进行, 检查步骤 步骤1配置成专线 先把报678错误的pppoe用户配置成专线方式，做ping测试来确定物理链路是通的。 即使物理链路不存在问题，但报678时，可能是pppoe报文在某环节丢失。,trouble shooting, 步骤2 在BAS上做pppoe报文分析 如果原来是动态pppoe配置，先将故障pppoe用户做成静态vlan配置 Profile pppoe-test ip unnumbered loopback 0 ppp authentication pap pppoe log pppoeControlPacket ppp log ppp packet ppp log pppstatemachine ! interface gigabitEthernet 1/0/1.6950435 svlan id 695 435 svlan ethertype 8100 pppoe pppoe autoconfigure pppoe profile any pppoe-test ！ 然后让故障用户现场拨号，在bas进行观察 E320#sh pppoe interface gigabitEthernet 1/0/1.6950435 PPPoE interface GigabitEthernet 1/0/1.6950435 is operStatusUp (dynamic) PPPoE interface GigabitEthernet 1/0/1.6950435 has max sessions = 5 PPPoE interface GigabitEthernet 1/0/1.6950435 MTU 1494 PPPoE interface GigabitEthernet 1/0/1.6950435 has no acName set PPPoE interface GigabitEthernet 1/0/1.6950435 has 1 active connections, out of 1 configured subinterfaces Attached QoS profile: qos-subscriber-face ethernet GigabitEthernet1/0/1 No baseline has been set PPPoE Statistics Counters: PADI received 1 PADI transmitted 0 PADO received 0 - PADO transmitted 1,如果PADI为0，就说明用户pppoe请求报文没传到BAS，故障点基本就是出在下层。 前提PADI为1，PADO为0时，则说明BAS出现问题，没有回应PADI报文。,trouble shooting,PADR received 1 PADR transmitted 0 PADS received 0 PADS transmitted 1 PADT received 1 PADT transmitted 0 PADM received 0 PADM transmitted 0 PADN received 0 PADN transmitted 0 PAD packets received 2 PAD packets transmitted 2 Invalid PAD Packets: Invalid Version 0 Invalid PAD Code 0 Invalid PAD Tags 0 Invalid PAD Tag length 0 Invalid PAD Type 0 Invalid PADI Session 0 Invalid PADR Session 0 Invalid PAD packet length 0 Invalid PAD packets Total Invalid PAD packets 4 Ingress Policed Packets 0 Egress Policed Packets 0 Insufficient Resources 0, PADT可以是用户发出，也可能是bas发出。, invalid PAD出现，则可能是客户端拨号软件设置或者网卡有问题。, Insufficient 则表示可能板卡或整机资源耗尽导致,trouble shooting, 根据pppoe报文的分析结果，采取以下步骤 BAS上各类PPPOE报文全为0 如果在E320上没有看到上述报文，则对汇聚交换机8512与E320直连的端口进行镜像抓包分析。PPPOE协议开始，客户端广播一个发起分组(PADI)，注意PADI为广播报文。 镜像示图：,镜像抓包完毕后，基本就能分析出报文丢在何处！,trouble shooting, E320上抓到的完整pppoe-ppp报文 baseline log E320#sh log data category pppoeControlPacket severity 7 de DEBUG 01/14/2010 16:11:46 pppoeControlPacket (interface ): PADI rx from 00e0.4dae.22ad, length 16, captured length 68, empty service name DEBUG 01/14/2010 16:11:46 pppoeControlPacket (interface ): PADO tx to 00e0.4dae.22ad, length 60, captured length 88, empty service name DEBUG 01/14/2010 16:11:46 pppoeControlPacket (interface ): PADR rx from 00e0.4dae.22ad, length 60, captured length 88, empty service name DEBUG 01/14/2010 16:11:46 pppoeControlPacket (interface ): PADS tx to 00e0.4dae.22ad, length 60, captured length 88, connection made using session id 14731 on sub interface 1 sh log data category ppppacket severity 7 de DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.00, tx lcp confReq , id = 229, length = 18, mru =1492, authentication = pap, magicNumber = 0x1e55816b DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.00, rx lcp confReq, id = 219, length = 14, mru =1492, magicNumber = 0x051cfd8c DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.00, tx lcp confAck, id = 219, length = 14, mru =1492, magicNumber = 0x051cfd8c DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.01, rx lcp confAck, id = 229, length = 18, mru =1492, authentication = pap, magicNumber = 0x1e55816b DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.01, rx lcp echoReq , id = 0, length = 8, magic =0x051cfd8c DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.01, tx lcp echoResP , id = 0, length = 8, magic =0x1e55816b DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.01, rx pap authReq, id = 1, length = 24, peerId length = 10, peerId = lsa5472359 6c 73 61 35 34 37 32 33 35 39, passwd length = 8, passwd = 16881988 31 36 38 38 31 39 38 38 DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.20, tx pap authAck, id = 1, length = 5, message,length = 0 DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.21, rx ipNcp confReq, id = 12, length = 22,ipAddress = 0.0.0.0, primaryDns = 0.0.0.0, secondaryDns = 0.0.0.0,trouble shooting,DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.21, tx ipNcp confNak, idj1 = 12, length = 22,ipAddress = 125.68.154.236 , primaryDns = 218.6.200.139, secondaryDns = 218.89.0.116 DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.21, rx ipNcp confReq, id = 13, length = 22,ipAddress = 125.68.154.236, primaryDns = 218.6.200.139, secondaryDns = 218.89.0.116 DEBUG 01/14/2010 16:11:46 pppPacket (interface GigabitEthernet2/1.101090299.113): time: 0.21, tx ipNcp confAck, id = 13, length = 22,ipAddress = 125.68.154.236, primaryDns = 218.6.200.139, secondaryDns = 218.89.0.116,trouble shooting, 镜像后，报文没发现丢失，则问题出在bas上 我们要通过shell里面去观察动态vlan生成、pppoe报文响应信息等综合判断是板卡故障，还是软件BUG造成。,1.2、718问题,718错误含义： 等待远程计算机有效响应的连接超时。 PPP 会话已启动，但由于远程计算机在适当的时间内没有响应而中断。这可能是由于线路质量太差或是由于服务器的问题而导致的。 检查配置，查看配置是否正确和没有遗漏。 在E320上做test测试，确认与radius通信是否正常。 ERX320#test aaa ppp ncscjjf 654321 bas资源不够也会造成报718错误,1.3、691问题, 出现691，基本与radius有关系。 测试步骤 首先在bas做test测试，如果radius上无法通过，并返回message错误信息，则在radius服务器上进行相应检查。 在BAS上带源地址ping radius服务器，看是否能ping通。 如果上述二项都正常，就是需要开debug查看bas与radius之间的报文,trouble shooting,2、L2tp业务trouble shooting, LAC domain map 的配置检查 E320#sh aaa domain-map Domain: aohua.gx; auth-router-name: default; ip-router-name: default; ipv6-router-name: default Tunnel Tunnel Tunnel Tunnel Tunnel Tunnel Tag Tunnel Peer Source Type Medium Password Id - - - - - - - 3 124.226.192.4 l2tp ipv4 aohua aohua Tunnel Tunnel Tunnel Tunnel Tunnel Client Server Tunnel Max Virtual Tag Name Name Preference Sessions Tunnel RWS Router - - - - - - - 3 aohua 2000 0 system chooses Tunnel Tunnel Tunnel Tx Tunnel Failover Switch Speed Tag Resync Profile Method - - - - 3 not set,查看配置是否正确 也可用show conf查看, 检查radius 的tunnel属性 E320# test aaa ppp blza45021102fca2.gx 5129813 Authentication Grant with Tunnel Attributes * user attributes * idle Timeout - 0 session Timeout - 0 accounting Timeout - 2700,trouble shooting,Tunnel Set - 1 Tunnel Tag set 0 domain-map中各隧道的标识 Tunnel Type set 3 隧道类型 3表示是L2TP Tunnel Medium set 1 1表示IPV4 Tunnel peer set - 202.103.230.150 Tunnel Password set - gxfcadsl Tunnel Router context - default Tunnel Hostname set - adsl Tunnel calling number - atm 2/0.42:100.110#184549476#this is a description#speed:UBR:12000#pppoe 12:34:56:78:9a:bc#,查看LAC上tunnel的状态 E320#sh l2tp tunnel L2TP tunnel 3/GXYLYBZhongXin is Up with 31 active sessions L2TP tunnel 5/ylcaizhengju is Up with 17 active sessions L2TP tunnel 15/222.216.27.1 is Up with 80 active sessions L2TP tunnel 16/222.216.27.2 is Up with 93 active sessions L2TP tunnel 455/yljiuqi is Up with 4 active sessions ！,表明tunnel的名称, 查看radius up地址和router-id是否一致 E320#show ip IP Router Id: 1.1.1.1 Administrative Router Id:1.1.1.1 Router Name: default Default TTL: 127 Reassemble Timeout: 30 SA Validate Trap: false,E320#show radius update-source-addr 1.1.1.1 E320#ping 1.1.1.1 source address 222.217.176.36 Sending 5 ICMP echoes to 116.11.186.51, timeout = 2 sec. ! Success rate = 100% (5/5), round-trip min/avg/max = 0/0/1 ms 如果LNS是禁ping的，需要做trace。,trouble shooting,3、Radius的trouble shooting, 首先查看radius的状态 E320#baseline radius E320#sh radius authentication statistics delta RADIUS Authentication Statistics - Statistic 61.139.30.35 61.139.81.3 61.139.81.6 - - - - UDP Port 1812 1812 1812 Round Trip Time 0 0 7 Access Requests 425 0 0 Rollover Requests 0 0 0 Retransmissions 0 0 0 Access Accepts 33 0 0 Access Rejects 392 0 0 Access Challenges 0 0 0 Malformed Responses 0 0 0 Bad Authenticators 0 0 0 Requests Pending 0 0 0 Request Timeouts 0 0 0 Unknown Responses 0 0 0 Packets Dropped 0 0 0 Statistics baseline set FRI JAN 22 2010 15:50:12 UTC, 有主备radius服务器设置 UDP 端口采用默认1812 access Requests为bas收到radius认证请求 Retransmissions为重传计数，radius与bas之间路由不通时，可能出现 正常情况下，rejects报文比accepts报文多 如果出现bad auth表明key可能配置出错 requests pending要求重传,trouble shooting,E320#show radius accounting statistics delta RADIUS Accounting Statistics - Statistic 61.139.30.35 61.139.81.3 61.139.81.6 - - - - UDP Port 1813 1813 1813 Round Trip Time 1 0 1 Requests 6043 0 0 Start Requests 1022 0 0 Interim Requests 4021 0 0 Stop Requests 1000 0 0 Reject Requests 0 0 0 Rollover Requests 0 0 0 Retransmissions 0 0 0 Responses 6043 0 0 Start Responses 1022 0 0 Interim Responses 4021 0 0 Stop Responses 1000 0 0 Reject Responses 0 0 0 Malformed Responses 0 0 0 Bad Authenticators 0 0 0 Requests Pending 0 0 0 Request Timeouts 0 0 0 Unknown Responses 0 0 0 Packets Dropped 0 0 0 Statistics baseline set FRI JAN 22 2010 15:50:12 UTC, requests与responses相对，是E320发给radius和收到radius报文总和。 start报文是开始计费的初始报文 interim报文是计费的update报文 stop报文是计费停止的报文, 查看E320上多少时间发送一次计费报文 E320#sh aaa accounting interval user-acct-interval 45 service-acct-interval 45,trouble shooting, 遇到认证不过故障时，经过查看radius状态能初步判断出故障因素；进一步更准确的确认故障成因，需要在E320上开radius的debug。 radiusSendAttributes 发送给radius的属性值 radiusAttributes 收到的radius属性值 radiusClient E320作为radius服务器的客户端，记录发送到radius的各类报文-认证、计费 打开debug E320(config)#log severity debug radiusAttributes E320(config)#log severity debug radiusSendAttributes E320(config)# log severity debug radiusClient 查看debug Baseline log E320# show log data ca radiusAttributes sev 7 de E320# show log data ca radiusSendAttributes sev 7 de E320# show log data ca radiusClient sev 7 de,trouble shooting, E320与radius之间的报文交互 示意图：,trouble shooting,E320的debug报文分析, 认证请求报文 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: ACCESS-REQUEST attributes (default) DEBUG 01/15/2010 23:43:48 radiusSendAttributes: username attr added: test6219149 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: acct-session-id attr added: erx atm 4/1/1.120840:3.307:0028577753 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: user-password attr added: DEBUG 01/15/2010 23:43:48 radiusSendAttributes: service-type attr added: 2 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: framed-protocol attr added: 1 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: pppoe-description (vsa) attr added: pppoe 00:90:96:30:ec:df DEBUG 01/15/2010 23:43:48 radiusSendAttributes: calling-station-id attr added: #E320#A41#3#307 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: connect-info attr added: speed:UBR DEBUG 01/15/2010 23:43:48 radiusSendAttributes: nas-port-type attr added: 16 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: nas-port attr added: 553845043 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: nas-port-id attr added: atm 4/1/1.120840:3.307 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: nas-ip-address attr added: 10.10.10.1 DEBUG 01/15/2010 23:43:48 radiusSendAttributes: nas-identifier attr added: E320, 计费请求报文 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: ACCOUNTING-REQUEST attributes (default) DEBUG 01/18/2010 11:24:08 radiusSendAttributes: acct-status-type attr added: 1 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: username attr added: CD87563401 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: event-timestamp attr added: 1263813848 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: acct-delay-time attr added: 0 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: nas-identifier attr added: BAS-CDZ1F-ERX710-1 DEBUG01/18/201011:24:08 radiusSendAttributes: acct-session-id attr added: erx GigabitEthernet 6/0.101010346:101-346:0324448074 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: nas-ip-address attr added: 222.212.128.1 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: service-type attr added: 2 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: framed-protocol attr added: 1 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: framed-compression attr added: 0 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: pppoe-description (vsa) attr added: pppoe 00:0a:eb:e6:f7:5d, 2、3,trouble shooting,DEBUG 01/18/2010 11:24:08 radiusSendAttributes: framed-ip-address attr added: 222.212.137.216 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: framed-ip-netmask attr added: 255.255.255.255 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: calling-station-id attr added: #BAS-CDZ1F-ERX710-1#E60#346 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: nas-port-type attr added: 15 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: nas-port attr added: 1610613082 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: nas-port-id attr added: GigabitEthernet 6/0.101010346:101-346 DEBUG 01/18/2010 11:24:08 radiusSendAttributes: acct-authentic attr added: 1, 认证返回报文 DEBUG 01/22/2010 22:14:19 radiusAttributes: USER ATTRIBUTES: (test2842580) DEBUG 01/22/2010 22:14:19 radiusAttributes: service type attr: 2 DEBUG 01/22/2010 22:14:19 radiusAttributes: framed protocol attr: 1 DEBUG 01/22/2010 22:14:19 radiusAttributes: session time attr: 172795 DEBUG 01/22/2010 22:14:19 radiusAttributes: total eap message attr length = 0 DEBUG 01/22/2010 22:14:19 radiusAttributes: ingress policy name (vsa) attr: pro512K DEBUG 01/22/2010 22:14:19 radiusAttributes: egress policy name (vsa) attr: pro1M DEBUG 01/22/2010 22:14:19 radiusAttributes: virtual router name (vsa) attr: default, 认证拒绝报文 DEBUG 01/22/2010 22:15:08 radiusAttributes: REJECT: reply message attr: User dial-in so soon. DEBUG 01/22/2010 22:15:08 radiusAttributes: REJECT: reply message attr: TOO MANY CONNECTIONS. DEBUG 01/22/2010 22:15:08 radiusAttributes: REJECT: reply message attr: User dial-in so soon. DEBUG 01/22/2010 22:15:08 radiusAttributes: REJECT: reply message attr: User dial-in so soon. DEBUG 01/22/2010 22:15:08 radiusAttributes: REJECT: reply message attr: cant find user.,trouble shooting,4、网管不通时trouble shooting, 专线、网管的trouble shooting相同，基于IPOA的方式则不相同 trouble shooting步骤 如果ping不通时，先做以下二步 E320#show ip route x.x.x.x E320#show arp | in x.x.x.x 如果上述二项ok，则做ping测试 E320(config)#log severity debug icmpTraffic remote-ip-address x.x.x.x E320#ping x.x.x.x 10 ping完后，查看log E320#baseline log E320#show log data category icmpTraffic de 主要查看tx、rx情况。 由于debug的信息只能表示SRP对icmp进行了转发，必要时需要定义端口统计策略计数icmp报文，来确认E320是否已经发出icmp报文 先定义策略 ip classifier-list “account-packet“ icmp any any ip policy-list “account-packet“ classifier-group “account-packet“ forward classifier-group * filter ! 应用到ip子端口下, 路由是否存在？ mac地址是否学习到？,trouble shooting,interface gigabitEthernet 0/0/5.20100 ! svlan id 2 100 svlan ethertype 8100 ip policy input “account-packet“ statistics enabled ip policy output “account-packet“ statistics enabled ip unnumbered loopback 24 ！ ping测试做完后，观察ip子接口统计 E320#show ip interface gigabitEthernet 0/0/5.20100 | begin IP policy IP policy input account-packet classifier-group account-packet entry 1 3 packets, 258 bytes forward classifier-group * 7 packets, 1062 bytes filter IP policy output account-packet classifier-group account-packet entry 1 3 packets, 258 bytes forward classifier-group * 0 packets, 0 bytes filter 查看二层、PC端的mac学习情况 需要注意MTU值、arp的时间等参数 E320上对arp的处理是不经过SRP的，只在线卡上ICFC。, 查看in和out的报文,trouble shooting,5、WLAN业务trouble shooting只讲E320上trouble shooting方法 WLAN系统由SRC、protal、radius、E320等多个系统组成， trouble shooting需要多个系统综合判断。这里，只介绍几个常见故障的trouble shooting方法。,5.1、 用户无法通过DHCP获得地址，从而打不开portal页面 请检查sscc的连接是否