海量系统日志采集解析实践

1、海量系统日志采集解析实践技术创新，变革未来01背景介绍02挑战及应对03采集手段04案例定制05现状简述灵活快速适应日志格式变化量大难以横向扩容昂贵按EPS 计费难以承受度量日志有效性监测02.挑战02.应对虚拟化集群支持横向扩容开源产品自我可控灵活按需解析、规范化格式富化增加监测点03.采集-Linuximuxsockimfileimpstatsomrelp03.采集-WindowsWinlogBeatPacketBeat FileBeatLogstash03.采集-DB/APIJDBCRestKAFKA03. 采集-SF TPSFTPShellPython Bat/Powershell03

2、.采集-过滤/转换/富化原始日志负载均衡过滤/转换/富化初级原始日志消息队列04. 案例定制需求分析解析运营04. 案例定制V 0需求：G t ab中代码拉取情况监测分析：G t ab中代码拉取途径HTTP/SSH,G t ab有多种类型日志文件,ap _son og、app cat on_son og、g t_son og、 g t ab-she og中包含相应拉取代码的日志信息解析：解析ap _son og、app cat on_son og、g t_son og、g t ab-she og,至少将相关仓库、地址、账号、SSHKey- d解 析,富化地址相关的网段信息运营：源网段设备拉取非

3、相关业务仓库的行为 V 2 0需求：离职人员离职前拉取代码情况 分析：从人力系统获得离职人员信息解析：人员拉取数据的事件及关键字段抽取入报表引擎数据库 运营：按需提供长期报表功能method: GET,path: /xxxx,format: */*,controller: Projects:GitHttpController, time: 2020-07-22T04:38:19.394Z,params: key: service,value: git-upload-pack,key: namespace_id, value: xxxxx,key: project_id,value: xxxxx

4、.git,remote_ip: xx.xx.xx.xx, user_id: xxxx, username: xxxx,ua: git/2.21.0 (Apple Git-122.2)”04. 案例定制原始日志初级富化日志检索timestamp: 2020-07-22 12:38:19 +0800,timegenerated: 2020-07-22 12:38:19 +0800, fromhost-ip: ”xxx. xxx. xxx. xxx , myhostname: xxx. xxx. xxx. xxx. xxx ,syslog-tag: duction_json, inputname:

5、imrelpxxxx, message: xxxxxxxxx04. 案例定制富化数据xx: xx网段name: GET,sourceAddress: xxx.xxx.xxx.xxx,sourceZoneURI: xx-xx.xx.xx.xx_xx.xx.xx.xx_xx产品部, requestMethod: GET,requestClientApplication: git/2.21.0 (Apple Git-122.2), deviceReceiptTime: 1595392699394, destinationUserName: xxxx,deviceSeverity: 200, requ

6、estUrl: /xxxx,deviceCustomString1: git-upload-pack, deviceCustomString2: xxxxx, deviceCustomString3: xxxxx,deviceEventCategory: duction_json, deviceFacility: Projects:GitHttpController, fileType: */*,关联分析method: GET,path: /xxxx,format: */*,controller: Projects:GitHttpController, time: 2020-07-22T04:

7、38:19.394Z,params: key: service,value: git-upload-pack,key: namespace_id, value: xxxxx,key: project_id,value: xxxxx.git,remote_ip: xx.xx.xx.xx, user_id: xxxx, username: xxxx,ua: git/2.21.0 (Apple Git-122.2)”原始日志关联数据仓库/部门映射表04. 案例定制name: GET,sourceAddress: xxx.xxx.xxx.xxx, sourceZoneURI: xxxx 产 品 部 ,

8、 requestMethod: GET, requestClientApplication: git deviceReceiptTime: 1595392699394, destinationUserName: xxxx, deviceSeverity: 200,requestUrl: /xxxx,deviceCustomString1: git-upload-pack, deviceCustomString2: xxxxx, deviceCustomString3: xxxxx, deviceEventCategory: duction_json, deviceFacility: Proje

9、cts:GitHttpController, fileType: */*,规范日志sourceAddress: xxx.xxx.xxx.xxx, sourceZoneURI: xxxx 产 品 部 , destinationUserName: xxxx, deviceCustomString1: git-upload-pack, deviceCustomString2: xxxxx按需定时 汇总报表05. 现状日志处理架构原始日志过滤转换富化初级富化日志初级富化日志队列归档解析富化数据规范日志队列抽取关联数据实时报警长期报表关联HDFS模式发现列式数据库事件运营设备类型设备信息网络设施华为/华

10、三/锐捷/NTO/安全SD-WAN基础设施ESXi/VDI/vCSA/TACACS/FreeIPA CMD/BIND/MS DHCP/MS DNS操作系统Linux操作系统日志/Linux Auditd日志Windows 安全/系统/Sysmon/Powershell安全设备网神防火墙/天眼/天擎EDR/天擎DLP/天空卫士DLP/天擎邮件沙箱/椒图/堡垒机/零信任/行为感 知分析系统/安全DNS/奇安信ID/安域/奇安信VPN/Suricata/Opencanary/OSQuery基础应用Apache Access Log/Nginx Access Log/IIS/Confluence/Jira/Jenkins/Gitlab/SVN/Sonar/artifactory/Exchange Protocol Logging/Exchange TraceL