飞塔防火墙_诊断

1、 Copyright Fortinet Inc. All rights reserved. 14 May 2022FortiGate IITroubleshootingFortiGate 5.2.12目标识别网络常规行为监控非正常行为如流量突发或非典型性协议Troubleshoot物理和逻辑网络接口理解会话表使用“diagnose debug flow” 来对流量流向进行排错 对资源使用问题进行排错, 如当防病毒和IPS打开时高CPU或高内存占用测试没有保存到flash的OS3在任何问题发生之前定义正常行为(基线):CPU 使用率Memory 使用率流量等级流量如何走向(流量)使用了哪些协议和

2、TCP/UDP 端口流量模式和分布Why?如果你知道什么是正常流量, 识别非正常流量会更容易NowBaseline (Average)Normal RangeAbnormal4网络图为何需要网络图? 没有网络图，解释和分析复杂网络是困难且耗时的物理图包含所有物理网络接口, 连线和端口对 Layer 1/2/3 的问题很有效逻辑图包含路由器, 逻辑设备(VDOMs)和UTM对Layer 3+的问题很有效2001:db8:b108port2/24port4/27port/8port35监控数据流 & 资源使用情况获取正常的网络数据 在发生

3、问题之前不正常的行为非常难发现 除非知道什么是正常的CPU使用率RAM使用率允许通过的应用入和出的带宽工具SNMPAlert emailLogging / SyslogFortiAnalyzer或者第三方SIEM(system information & event management)Dashboard / get system statusNormalTraffic spikes6SNMPAllowed source of queries7通过SNMP获取事件通知 trapDestination触发FortiGate t发送SNMP消息的事件8# get sys statusVersio

4、n: FortiGate-VM64 v5.2.0,build0589,140613 (GA)Virus-DB: 22.00856(2014-09-24 05:33)Extended DB: 1.00000(2012-10-17 15:46)IPS-DB: 5.00549(2014-09-23 00:49)IPS-ETDB: 0.00000(2001-01-01 00:00)Serial-Number: FGVM040000025212Botnet DB: 1.00736(2014-08-24 10:18)License Status: ValidVM Resources: 1 CPU/4 al

5、lowed, 969 MB RAM/6144 MB allowedBIOS version: 04000002Log hard disk: AvailableHostname: STUDENTOperation Mode: NATCurrent virtual domain: rootMax number of virtual domains: 10Virtual domains status: 1 in NAT mode, 0 in TP modeVirtual domain configuration: disableFIPS-CC mode: disableCurrent HA mode

6、: standaloneBranch point: 589Release Version Information: GAFortiOS x86-64: YesSystem time: Thu Oct 9 00:26:54 2014# get sys perf statCPU states: 2% user 15% system 0% nice 83% idleCPU0 states: 2% user 15% system 0% nice 83% idleMemory states: 44% usedAverage network usage: 542 kbps in 1 minute, 105

7、0 kbps in 10 minutes, 512 kbps in 30 minutesAverage sessions: 7 sessions in 1 minute, 5 sessions in 10 minutes, 5 sessions in 30 minutesAverage session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutesVirus caught:

8、0 total in 1 minuteIPS attacks blocked: 0 total in 1 minuteUptime: 0 days, 0 hours, 19 minutes系统信息 & 资源使用情况9# diagnose firewall statistic showgetting traffic statistics.Browsing: 328 packets, 132562 bytesDNS: 797 packets, 127917 bytesE-Mail: 0 packets, 0 bytesFTP: 0 packets, 0 bytesGaming: 0 packets

9、, 0 bytesIM: 0 packets, 0 bytesNewsgroups: 0 packets, 0 bytesP2P: 0 packets, 0 bytesStreaming: 0 packets, 0 bytesTFTP: 0 packets, 0 bytesVoIP: 0 packets, 0 bytesGeneric TCP: 1098554 packets, 817573554 bytesGeneric UDP: 1490 packets, 210976 bytesGeneric ICMP: 0 packets, 0 bytesGeneric IP: 6 packets,

10、192 bytes# diagnose hardware deviceinfo nic port1Name: port1Driver: e1000Version: 5.1.13k2 NAPIFW version: N/ABus: 00:11.0Memory: 0 xfeb80000 - 0 xfeba0000Base address: 0 x1400Interrupt: 18Hwaddr: 00:0c:29:95:8c:faPermanent Hwaddr:00:0c:29:95:8c:faState: upLink: upMtu: 1500Supported: auto 10half 10f

11、ull 100half 100full 1000fullAdvertised: auto 10half 10full 100half 100full 1000fullSpeed: 1000fullAuto: enabledRx packets: 136154Rx bytes: 10901815Rx compressed: 0Rx dropped: 0Rx errors: 0 Rx Length err: 0 Rx Buf overflow: 0 Rx Crc err: 0 Rx Frame err: 0 Rx Fifo overrun: 0 Rx Missed packets: 0Tx pac

12、kets: 1611Tx bytes: 257565.Multicasts: 0Collisions: 0带宽利用率，系统崩溃和错误10其他工具CLI get system status get system performance status diagnose sys top diagnose sys top-summary diagnose hardware sysinfo memory diagnose hardware sysinfo shm diagnose netlink device list diagnose hardware deviceinfo nic port1 dia

13、gnose firewall statistics show .DashboardSNMP trapsAlert emailLogs11# diagnose hardware deviceinfo nic port1Description :FortiASIC NP6 AdapterDriver Name :FortiASIC Unified NPU DriverName :np6_2PCI Slot :8d:00.0irq :58Board :FGT3700DSN :NP6KR44613000276Major ID :2Minor ID :0lif id :0lif oid :156netd

14、ev oid :156netdev flags :1203Current_HWaddr :08:5b:0e:4a:2e:e4Permanent_HWaddr:08:5b:0e:4a:2e:e4phy name :np6_2_0bank_id :255phy_addr :0 x20lane :0sw_port :51sw_np_port (cat)vid_phy6 :0 x000 x000 x0b0 x000 x000 x00vid_fwd6 :0 x000 x000 x000 x000 x000 x00oid_fwd6 :0 x000 x000 x000 xcc0 x000 x00= Link

15、 Status =Admin :upnetdev status :downautonego_setting:1link_setting :1link_speed :40000link_duplex :1Speed :0Duplex :Fulllink_status :Downrx_link_status :0int_phy_link :0local_fault :0local_warning :0remote_fault :0= Counters =Rx Pkts :0Rx Bytes :0Tx Pkts :0Tx Bytes :0Host Rx Pkts :0Host Rx Bytes :0

16、Host Rx dropped :0Host Tx Pkts :4Host Tx Bytes :198Host Tx dropped :0sw_rx_pkts :0sw_rx_bytes :0sw_tx_pkts :0sw_tx_bytes :0sw_np_rx_pkts :4sw_np_rx_bytes :272sw_np_tx_pkts :0sw_np_tx_bytes :0物理层/数据链路层的Troubleshooting12网络层的Troubleshooting：路由# execute ping-options ?data-size 定义数据包的大小,以bytes为单位df-bit 在

17、IP头里设置 DF 位interval 两个ping直接的间隔时间,以秒为单位pattern 十六进制格式, e.g. 00ffaabbrepeat-count 重复ping多少次source auto | timeout 定义多少秒后timeouttos IP的服务类型ttl 存活时间 time-to-live.validate-reply 有效的reply数据.view-settings 查看ping的当前设置# execute ping # execute traceroute | 13网络层的Troubleshooting：会话1.清空之前的过滤条件# diagnose sys ses

18、sion filter clear2.设置过滤条件# diagnose sys session filter ?dport destination portdst destination IP addresspolicy policy idsport source portsrc source ip address3.列出所有匹配过滤条件的会话# diagnose sys session list4.清空所有匹配过滤条件的会话# diagnose sys session clear14会话表：TCPsession info: proto=6 proto_state=65 duration=3

19、expire=9 timeout=3600 flags=00000000 sockflag=00000000 sockport=443 av_idx=9 use=5origin-shaper=guarantee-100kbps prio=2 guarantee 12800Bps max 134217728Bps traffic 13895Bpsreply-shaper=guarantee-100kbps prio=2 guarantee 12800Bps max 134217728Bps traffic 13895Bpsper_ip_shaper=ha_id=0 policy_dir=0 tu

20、nnel=/state=redir local may_dirty ndr npu nlb os rs statistic(bytes/packets/allow_err): org=864/8/1 reply=2384/7/1 tuples=3orgin-sink: org pre-post, reply pre-post dev=7-6/6-7 gwy=/hook=post dir=org act=snat 10:57999-9:443(6:57999)hook=pre dir=rep

21、ly act=dnat 9:443-6:57999(10:57999)hook=post dir=reply act=noop 9:443-10:57999(:0)pos/(before,after) 0/(0,0), 0/(0,0)misc=0 policy_id=1 id_policy_id=0 auth_info=0 chk_client_info=0 vd=0serial=0008b037 tos=ff/ff ips_view=1 app_list=2000 ap

22、p=24534dd_type=0 dd_mode=0per_ip_bandwidth meter: addr=10, bps=4872npu_state=00000000npu info: flag=0 x00/0 x00, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0/0连连接状接状态态流量整形流量整形数据包数据包统计统计剩余剩余TTLNAT硬件加速硬件加速协议协议目的端口目的端口会会话处话处理理15会话表：协议proto=6服务代码在IP头中常见的代码1 = ICMP6 = TCP17 = UDP13

23、2 = SCTPIPv4 Header16传输层的Troubleshooting: TCP状态proto_state=05总是两位数字第一位数字 = 客户端的会话状态（没有代理则是0）第二位数字 = 服务器端的会话状态StateValueExpiry Timer (default)NONE010 sESTABLISHED13600 sSYN_SENT2120 sSYN & SYN/ACK360 sFIN_WAIT4120 sTIME_WAIT5120 sCLOSE610 sCLOSE_WAIT7120 sLAST_ACK830 sLISTEN9120 s17传输层的Troubleshootin

24、g: UDP & ICMP 状态虽然UDP 是一个无状态协议, FortiGate 仍会有两个不同的 “proto_state” 值:State Value未看到UDP回应 00看到UDP回应01ICMP 无状态proto_state 一直标记为0018传输层的Troubleshooting: SCTP状态StateValueExpiry Timer (default)SCTP_S_NONE060 sSCTP_S_ESTABLISHED13600 sSCTP_S_CLOSED210 sSCTP_S_COOKIE_WAIT35 sSCTP_S_COOKIE_ECHOED410 sSCTP_S_S

25、HUTDOWN_SENT530 sSCTP_S_SHUTDOWN_RECD630 sSCTP_S_SHUTDOWN_ACK_SENT73 sSCTP_S_MAX8n/a19会话表：会话处理标识state=log shape may_dirty 并不通用 如果会话被卸载到ASIC芯片上，则不一定代表是现在的状态hardware accelerationFlagMeaninglogSession is being loggedlocalSession is to/from local stackextSession is created by a firewall session helperma

26、y_dirtySession is created by traffic hitting a policy. ndrSession will be checked by IPS signaturendsSession will be checked by IPS anomalybrSession is being bridged (TP mode)npuSession is possible to be offloaded to NPUwccpSession is handled by WCCPnpdSession cannot be offloaded to NPUdirtyNext pac

27、ket in original direction will be revalidated against policyredirSession is being processed by an application layer proxyauthedSession was successfully authenticatedauthSession is requires (or required) authenticationsrc-visSession is being scanned for device detection purposes20会话表: 连接自动删除会话超时expir

28、e=89 timeout=3600不活跃的会话当两个值都为0时TCP连接被拆除FIN, FIN/ACK, ACKTCP连接超时tcp-halfclose-timer: FIN WAIT and CLOSE WAITtcp-half-open-timer: SYN SENT and SYN & SYN/ACKtcp-timewait-timer: TIME WAITudp-idle-timer21高级抓包选项#diag sniffer packet 当抓取了这个数目的报文时自动停止抓包 修改时间戳的格式a 绝对UTC时间l 当地时间22高级抓包选项：输出 # diag sniff packet

29、any icmp 4 interfaces=any filters=icmp 2.101199 wan2 in 10 - : icmp: echo request 2.101400 wan1 out 6 - : icmp: echo request 2.123325 wan1 in - 6: icmp: echo reply 2.123500 wan2 out - 10: icmp: echo reply 4 packets received by

30、 filter 0 packets dropped by kernel # diag sniff packet any icmp 4 3 l interfaces=any filters=icmp 2014-11-14 10:28:19.769989 wan2 in 10 - : icmp: echo request 2014-11-14 10:28:19.770143 wan1 out 6 - : icmp: echo request 2014-11-14 10:28:19.792325 wan1 in

31、- 6: icmp: echo reply 3 packets received by filter 0 packets dropped by kernel报文数量时间戳23诊断系统反应过慢高高CPU使用率使用率高内存使用率高内存使用率上一个开启的功能是什么？每次开启一个功能快速诊断CPU使用率有多高，为什么？# get system performance status# diagnose sys top 124高CPU占用率的Troubleshooting: get sys perf stat# get system performance statusCPU sta

32、tes: 4% user 13% system 0% nice 83% idleCPU0 states: 3% user 13% system 0% nice 84% idleCPU1 states: 5% user 13% system 0% nice 82% idleCPU2 states: 2% user 13% system 0% nice 85% idleCPU3 states: 6% user 13% system 0% nice 81% idleMemory states: 19% usedAverage network usage: 12740 kbps in 1 minute

33、, 3573 kbps in 10 minutes, 1077 kbps in 30 minutesAverage sessions: 118 sessions in 1 minute, 11 sessions in 10 minutes, 40 sessions in 30 minutesAverage session setup rate: 11 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 1 sessions per second in last 30 minutesVir

34、us caught: 3 total in 1 minuteIPS attacks blocked: 64 total in 1 minuteUptime: 60 days, 9 hours, 58 minutesCPU使用率网络使用率内存使用率25高CPU使用率：临时bypass一些检查进程n可以暂时bypass一些检查进程# diagnose test application ipsmonitor 5全局bypass更容易,之后可以调整策略来确认问题无需检查的任务继续运行n CPU使用率在bypass这些进程后是否降低了？n恢复某个检查进程：# diagnose test applicat

35、ion ipsmonitor 526内存诊断进程的内存使用率# get system performance status# diag sys top-summaryFortiOS的内存使用率，不是某一个进程的# diagnose hardware sysinfo mem# diagnose hardware sysinfo slab27# diagnose sys top-summary CPU | 38.4% Mem | 54.0% 1009M/1841M Processes: 20 (running=1 sleeping=86) PID RSS CPU% MEM% FDS TIME+ N

36、AME * 72 32M 34.2 1.7 11 00:03.39 httpclid x5 95 11M 1.9 0.6 20 53:07.83 cw_wtpd 40 23M 1.2 1.3 24 03:02.60 httpsd x5 1173 27M 0.0 1.5 10 00:02.82 pyfcgid x4 36 10M 0.0 0.5 88 00:47.75 zebos_launcher x12 37 9M 0.0 0.5 9 00:00.23 uploadd 38 15M 0.0 0.8 41 01:52.19 miglogd 39 9M 0.0 0.5 5 00:01.41 kmi

37、glogd 46 25M 0.0 1.4 821 01:47.98 proxyd x6 47 10M 0.0 0.5 7 00:00.12 wad_diskd 51 12M 0.0 0.7 16 00:02.72 scanunitd x3 53 61M 0.0 3.3 16 00:15.14 ipsmonitor x2 57 9M 0.0 0.5 7 00:00.13 merged_daemons 69 13M 0.0 0.7 18 00:34.20 urlfilter 在diag sys top 中RAM复杂 交叉的进程会使条目众多 交叉的进程中共享数据# diagnose sys topRun Time: 11 days, 3 hours and 29 minutes0U, 0S, 10I; 500T, 345F, 78KF thttp 48 S 0.0 4.4 httpsd 74 S 0.0 3.4 httpsd 54 S 0.0 3.4 cmdbsvr 23 S 0.0 3.4 httpsd 18618 S 0.0 2.9 httpsd 18645 S 0.0 2.9 httpsd 18643 S