juniperSRX命令行排障.ppt

SRX ADVANCED TROUBLESHOOTING,AGENDA,Flow traceoptions NAT Application Layer Gateway (ALG) IPSec VPN Chassis Cluster Core-dumps UTM,Flow traceoptions,Flow traceoptions,Configured under “security flow” hierarchy: Traces for flow processing Packet filters Default action = permit “security datapath-debug” for SRX5000 and SRX3000 in JUNOS 10.0 and later End-to-end trace: Ingress IOC Ingress flow Flow processing Egress flow Egress IOC Packet dump Action-profile Packet filters Default action = deny,Configuration Example,rootcumin# show security flow traceoptions file flow-trace size 1m files 2 world-readable; flag basic-datapath; packet-filter incoming source-prefix 1.1.1.0/24; destination-prefix 4.4.4.0/24; packet-filter outgoing source-prefix 4.4.4.0/24; destination-prefix 1.1.1.0/23; ,primary:node1edit rootcumin# show security datapath-debug traceoptions file new-debug size 10m files 3 world-readable; action-profile new-debug preserve-trace-order; event np-egress count; module flow flag all; packet-filter ping action-profile new-debug; protocol icmp; source-prefix 1.1.1.0/24; destination-prefix 4.4.4.0/24; ,Flow traceoptions,Configure packet filters for desired traffic Check output with “show log ” Use trim and match options to show output Observe time stamp Use “preserve-trace-order” for SRX5000 and SRX3000 to avoid traces out of order due to multithreaded processing Useful to troubleshooting flow processing: Route Policy Session NAT ALG IDP UTM IPSec VPN Use with caution - flow tracing is CPU intensive and may affect operation,JTAC Troubleshooting Recommendation,Flow traceoptions,1) Pull Packet from queue 2) Police Packet 3) Filter Packet 4) Lookup Session: 4.a) No Match = First Path a) FW Screen Check b) Route Lookup c) Find Destination Zone d) Look-up Policy e) Allocate NAT f) Setup ALG vector g) Install Session 4.b) Match = Fast Path a) FW Screen Check b) TCP Checks c) NAT Translation g) ALG Processing 5) Filter Packet 6) Shape Packet 7) Transmit Packet,Flow processing,Event Scheduler,Per Packet Filters,Per Packet Policers,/,Shapers,Flow traceoptions,Problem: Traffic not passing through. Scenario: SRX240 standalone JUNOS 9.6R1 Description: user trying to access a protected server behind the firewall. User initiates the connection but it times out Suspicion is that SRX240 is dropping the traffic as the network addressing and protected server are properly configured.,Troubleshooting Example,Flow traceoptions,Commands used: monitor interface show security flow session summary show security flow session destination-prefix 28.1.1.2 show security flow session session-identifier show interface extensive Traceoptions enabled: set security flow traceoptions file flow-trace set security flow traceoptions file size 2m set security flow traceoptions file files 2 set security flow traceoptions file world-readable set security flow traceoptions flag basic-datapath set security flow traceoptions packet-filter incoming source-prefix 1.1.1.0/24 set security flow traceoptions packet-filter incoming destination-prefix 28.0.0.0/8 Logs checked: /var/log/flow-trace,Troubleshooting Example,Flow traceoptions,With “monitor interface ” its possible to see if traffic is really arriving the SRX And also, confirm if its leaving, and if on the right interface.,hankey Seconds: 19 Time: 19:39:57 Delay: 0/0/0 Interface: ge-0/0/10.0, Enabled, Link is Up Flags: SNMP-Traps Encapsulation: ENET2 Local statistics: Current delta Input bytes: 468 0 Output bytes: 178384 584 Input packets: 7 0 Output packets: 1304 4 Remote statistics: Input bytes: 35124 (68800 bps) 588 Output bytes: 13797 (0 bps) 0 Input packets: 461 (100 pps) 7 Output packets: 188 (0 pps) 0 Traffic statistics: Input bytes: 35592 588 Output bytes: 192181 584 Input packets: 468 7 Output packets: 1492 4 Protocol: inet, MTU: 1500,monitor interface ,Flow traceoptions,Traffic is arriving on the interface, but there is no session created for the flow were interested in.,roothankey# run show security flow session destination-port 8080 0 sessions displayed edit,show security flow session,Flow traceoptions,With flow traceoptions its possible to see how the packet is processed:,edit roothankey# run show log flow-trace | trim 42 | no-more 28.1.1.2/8080;6 matched filter incoming: packet 48 ipid = 51867, 428e721c - flow_process_pkt: (thd 3): flow_ctxt type 13, common flag 0x0, mbuf 0x428e7080 flow process pak fast ifl 72 in_ifp ge-0/0/10.0 ge-0/0/10.0:1.1.1.10/54090-28.1.1.2/8080, tcp, flag 2 syn find flow: table 0x49ea9da8, hash 60937(0xffff), sa 1.1.1.10, da 28.1.1.2, sp 54090, dp 8080, proto 6, tok 576 no session found, start first path. in_tunnel - 0, from_cp_flag - 0 flow_first_create_session flow_first_in_dst_nat: in , out dst_adr 28.1.1.2, sp 54090, dp 8080 chose interface ge-0/0/10.0 as incoming nat if. flow_first_rule_dst_xlate: DST no-xlate: 0.0.0.0(0) to 28.1.1.2(8080) flow_first_routing: call flow_route_lookup(): src_ip 1.1.1.10, x_dst_ip 28.1.1.2, in ifp ge-0/0/10.0, out ifp N/A sp 54090, dp 8080, ip_proto 6, tos 10 Doing DESTINATION addr route-lookup routed (x_dst_ip 28.1.1.2) from external (ge-0/0/10.0 in 0) to ge-0/0/8.0, Next-hop: 28.1.1.2 policy search from zone external- zone internal app 0, timeout 1800s, curr ageout 20s packet dropped, denied by policy packet dropped, policy deny. flow find session returns error. - flow_process_pkt rc 0x7 (fp rc -1),show log flow-trace,Flow traceoptions,Example of output without triming/matching: Detailed time stamps Cluster ID FPC/PIC/Thread ID (SRX5000 and SRX3000),() Nov 29 15:16:29 11:49:00.906432:CID-02:FPC-05:PIC-00:THREAD_ID-23:RT: flow_first_create_session Nov 29 15:16:29 11:49:00.906457:CID-02:FPC-05:PIC-00:THREAD_ID-23:RT: flow_first_in_dst_nat: in , out dst_adr 4.4.4.10, sp 0, dp 5034 Nov 29 15:16:29 11:49:00.906490:CID-02:FPC-05:PIC-00:THREAD_ID-23:RT: chose interface reth0.0 as incoming nat if. Nov 29 15:16:29 11:49:00.906514:CID-02:FPC-05:PIC-00:THREAD_ID-23:RT:flow_first_rule_dst_xlate: DST no-xlate: 0.0.0.0(0) to 4.4.4.10(5034) Nov 29 15:16:29 11:49:00.906548:CID-02:FPC-05:PIC-00:THREAD_ID-23:RT:flow_first_routing: call flow_route_lookup(): src_ip 1.1.1.10, x_dst_ip 4.4.4.10, in ifp reth0.0, out ifp N/A sp 0, dp 5034, ip_proto 1, tos 0 Nov 29 15:16:29 11:49:00.906591:CID-02:FPC-05:PIC-00:THREAD_ID-23:RT:Doing DESTINATION addr route-lookup Nov 29 15:16:29 11:49:00.906619:CID-02:FPC-05:PIC-00:THREAD_ID-23:RT: routed (x_dst_ip 4.4.4.10) from client (reth0.0 in 1) to reth3.0, Next-hop: 4.4.4.10 Nov 29 15:16:29 11:49:00.906656:CID-02:FPC-05:PIC-00:THREAD_ID-23:RT: policy search from zone client- zone vpn-ssg (),show log flow-trace,Flow traceoptions,Detailed information, including physical and logical interfaces, statistics, security counters,primary:node1edit rootcumin# run show interfaces reth4 extensive Physical interface: reth4, Enabled, Physical link is Up Interface index: 132, SNMP ifIndex: 124, Generation: 135 Link-level type: Ethernet, MTU: 1514, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 0 Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x4000 Current address: 00:10:db:ff:60:04, Hardware address: 00:10:db:ff:60:04 Last flapped : 2009-11-20 19:41:06 UTC (1w4d 15:21 ago) Statistics last cleared: Never Traffic statistics: Input bytes : 263294337 0 bps Output bytes : 18122614 0 bps Input packets: 3789769 0 pps Output packets: 91475 0 pps IPv6 transit statistics: Input bytes : 0 Output bytes : 0 Input packets: 0 Output packets: 0 Dropped traffic statistics due to STP State: Input bytes : 0 Output bytes : 0 Input packets: 0 Output packets: 0,show interface extensive,Flow traceoptions,Input errors: Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Giants: 0, Policed discards: 0, Resource errors: 0 Output errors: Carrier transitions: 0, Errors: 0, Drops: 0, MTU errors: 0, Resource errors: 0 Egress queues: 8 supported, 4 in use Queue counters: Queued packets Transmitted packets Dropped packets 0 best-effort 2812123 2812123 0 1 expedited-fo 0 0 0 2 assured-forw 0 0 0 3 network-cont 0 0 0 Logical interface reth4.0 (Index 72) (SNMP ifIndex 125) (Generation 137) Flags: SNMP-Traps 0x4004000 Encapsulation: ENET2 Statistics Packets pps Bytes bps Bundle: Input : 3882739 0 277518157 0 Output: 2812124 0 132390462 0 Link: ge-3/1/7.0 Input : 309229 0 23209057 0 Output: 0 0 0 0 ge-9/1/7.0 Input : 3573510 0 254309100 0 Output: 2812124 0 132390462 0,show interface extensive,Flow traceoptions,Security: Zone: vpn-ssg Allowed host-inbound traffic : any-service bfd bgp dvmrp igmp ldp msdp nhrp ospf pgm pim rip router-discovery rsvp sap vrrp Flow Statistics : Flow Input statistics : Self packets : 23475 ICMP packets : 0 VPN packets : 69506 Multicast packets : 0 Bytes permitted by policy : 4779676 Connections established : 83 Flow Output statistics: Multicast packets : 0 Bytes permitted by policy : 6482096 Flow error statistics (Packets dropped due to): Address spoofing: 0 Authentication failed: 0 Incoming NAT errors: 0 Invalid zone received packet: 0 Multiple user authentications: 0 Multiple incoming NAT: 0 No parent for a gate: 0 No one interested in self packets: 0 No minor session: 0 No more sessions: 0 No NAT gate: 0 No route present: 0 No SA for incoming SPI: 442 No tunnel found: 0 No session for a gate: 0 No zone or NULL zone binding 0 Policy denied: 0 Security association not active: 25776 TCP sequence number out of window: 0 Syn-attack protection: 0 User authentication errors: 0,show interface extensive,Flow traceoptions,Route Cause: after checking the configuration it showed that the policy was deactivated: Solution: Activate policy.,Troubleshooting Example,roothankey# show security policies from-zone external to-zone internal | display set set security policies from-zone external to-zone internal policy allow-mgt match source-address any set security policies from-zone external to-zone internal policy allow-mgt match destination-address any set security policies from-zone external to-zone internal policy allow-mgt match application any set security policies from-zone external to-zone internal policy allow-mgt then permit set security policies from-zone external to-zone internal policy allow-mgt then log session-init set security policies from-zone external to-zone internal policy allow-mgt then log session-close set security policies from-zone external to-zone internal policy allow-mgt then count deactivate security policies from-zone external to-zone internal,Flow traceoptions,Session is created.,Troubleshooting Example,roothankey# run show security flow session destination-port 8080 Session ID: 12095, Policy name: allow-mgt/8, Timeout: 12 In: 1.1.1.10/54139 28.1.1.2/8080;tcp, If: ge-0/0/10.0 Out: 28.1.1.2/8080 1.1.1.10/54139;tcp, If: ge-0/0/8.0 1 sessions displayed edit roothankey# run show security flow session session-identifier 12095 Session ID: 12095, Status: Normal Flag: 0x40000040 Policy name: allow-mgt/8 Source NAT pool: Null Maximum timeout: 1800, Current timeout: 2 Start time: 3200564, Duration: 17 In: 1.1.1.10/54139 28.1.1.2/8080;tcp, Interface: ge-0/0/10.0, Session token: 0x240, Flag: 0x4129 Route: 0xd0010, Gateway: 8.8.8.10, Tunnel: 0 Port sequence: 0, FIN sequence: 0, FIN state: 0, Out: 28.1.1.2/8080 1.1.1.10/54139;tcp, Interface: ge-0/0/8.0, Session token: 0x280, Flag: 0x32 Route: 0xf0010, Gateway: 28.1.1.2, Tunnel: 0 Port sequence: 0, FIN sequence: 0, FIN state: 0, 1 sessions displayed,Flow traceoptions,Flow traceoptions for session creation:,Troubleshooting Example,() Doing DESTINATION addr route-lookup routed (x_dst_ip 28.1.1.2) from external (ge-0/0/10.0 in 0) to ge-0/0/8.0, Next-hop: 28.1.1.2 policy search from zone external- zone internal app 0, timeout 1800s, curr ageout 20s flow_first_src_xlate: src nat 0.0.0.0(54139) to 28.1.1.2(8080) returns status 0, rule/pool id 0/0. choose interface ge-0/0/8.0 as outgoing phy if is_loop_pak: No loop: on ifp: ge-0/0/8.0, addr: 28.1.1.2, rtt_idx:0 policy is NULL (wx/pim scenario) sm_flow_interest_check: app_id 0, policy 8, app_svc_en 0, flags 0x2. not interested sm_flow_interest_check: app_id 1, policy 8, app_svc_en 0, flags 0x2. not interested flow_first_service_lookup(): natp(0x4d4d4920): app_id, 0(0). service lookup identified service 0. flow_first_final_check: in , out existing vector list 2-44ef6828. Session (id:12095) created for first pak 2 flow_first_install_session= 0x4d4d4920 make_nsp_ready_no_resolve() route lookup: dest-ip 1.1.1.10 orig ifp ge-0/0/10.0 output_ifp ge-0/0/10.0 orig-zone 9 out-zone 9 vsd 0 route to 8.8.8.10 Installing c2s NP session wing Installing s2c NP session wing flow got session. flow session id 12095 mbuf 0x42993280, exit nh 0xf0010 - flow_process_pkt rc 0x0 (fp rc 0), - Troubleshooting Traffic Flows and Session Establishment,nat,nat,Source NAT using an IP pool “address-persistent” option enabled One rule-set from zone trust to untrust One rule to translate all traffic Proxy-ARP configured to allow for hosts to respond to the NATed addresses Other NAT types: Destination Static,Configuration Example Source NAT,primary:node0 rootSRX5800-1 show configuration security nat source pool source_nat_pool address 10.1.1.0/24; address-persistent; rule-set first_rule_set from zone trust; to zone untrust; rule rule1 match source-address 0.0.0.0/0; then source-nat pool source_nat_pool; proxy-arp interface reth5.0 address 10.1.1.0/24; ,nat,Static & destination NAT are performed before security policies are applied Reverse static & source NAT are performed after security policies are applied,NAT Processing,nat,If a packet matches multiple rule-sets, the most specific match takes precedence Order of precedence (first destination then source) Interface Zone Routing-Instance Rules in a rule-set are evaluated in order,Rule-set Priorities,nat,Is source NAT using IP pool? Does the pool have enough addresses? Is port translation required? Is address persistent required? Is the rule-set applied to the correct interfaces/zones/routing-instances? Is the order of rules correct? Is proxy-ARP configured? Are there other NAT types being matched?,Troubleshooting Checklist,nat,Commands: show security nat source summary show security nat source rule show security nat source pool show security nat destination summary show security nat destination pool show security nat destination rule show security nat static rule show security flow session request support information Traceoptions (in combination with): Flow traceoptions set security flow traceoptions set security nat traceoptions flag all (SRX5000 and SRX3000 only) Datapath-debug JUNOS 10.0 and later set security datapath-debug set security nat traceoptions flag all Logs: (under /var/log/ directory): show log ,JTAC Troubleshooting Recommendation,nat,Shows details of all the pools and rules configured for source NAT.,show security nat source summary,primary:node1edit rootcumin# run show security nat source summary node 1 node1: - Total pools: 5 Pool Address Routing PAT Total Name Range Instance Address client-vpn 11.1.1.1-11.1.1.1 default yes 1 first-range 2.2.2.152-2.2.2.155 default yes 4 second-range 2.2.2.187-2.2.2.198 default yes 12 test-range 93.122.135.1-93.122.135.1 default yes 1 remote-site 4.4.4.22-4.4.4.22 default yes 1 Total rules: 4 Rule name Rule set From To Action one multicast client multicast interface black-zone first reth0.0 default white-pool white-zone server client first first blue-zone client-to-s reth3.0 reth4.0 test,nat,Shows details of each source NAT rule: “Translation hits” to check if there is traffic matching this rule,show security nat source rule,primary:node1edit rootcumin# run show security nat source rule node 1 windows-ssg node1: - source NAT rule: windows-ssg Rule-set: first Rule-Id : 2 From interface : reth0.0 To routing instance : default Match Source addresses : 1.1.1.11 - 1.1.1.11 Destination addresses : 4.4.4.10 - 4.4.4.10 Action : windows Persistent NAT type : N/A Inactivity timeout : 0 Max session number : 0 Translation hits : 3236,nat,Shows details of each source NAT pool: “Translation hits” to check if there is traffic using IPs from this pool “Single Ports” to check pool resource utilization E.g. there are 4 existing sessions using this pool,show security nat source pool,primary:node1edit rootcumin# run show security nat source pool windows node 1 node1: - Pool name : my_pool Pool id : 8 Routing ins