基于云计算技术的网络告警融合系统的设计与实现-英

1、基于云计算技术的网络告警融合分析系统的设计与实现李洪敏（1） 卢敏（1） 黄林（2） 张建平（1）（1）中国工程物理研究院总体工程研究所（四川绵阳 919 信箱 428 分箱 621900）（2）西南科技大学计算机学院摘 要：针对涉密网中安全设备产生的结构多样、数据庞大日志信息，将这些彼此独立的片断信息链接起来，重现整个网络攻击过程，发现攻击者的真正意图，是目前网络安全态势研究的重点和难点。本文基于以上需求，研究多源海量日志数据处理分析技术，构建层次型数据融合处理框架，实现从数据层到特征层，再到决策层的多源海量日志数据的融合处理，设计并实现基于云计算技术的网络告警数据分析系统。关键词： 云计算

2、；告警融合；架构设计中图分类号：TP393 文献标识码：AThe Design and Implement of A Network Alarm and Data Fusion Analysis System Based on Cloud ComputingLihongmin1 Lumin1 Huanglin2 Zhanjianping1(1.Institute of System Engineering, CAEP, Mianyang 621900,China；2.Computer College, SWUST，Miany0ng 621010,China）Abstract: The curr

3、ent security experts focus on a difficult study which is the process-reproduce of the whole network attack by linking the independent pieces of infinite information from various kinds of security equipment. Based on the above requirements, this paper designs and implements a network alarm and data a

4、nalysis system by researching on the analysis techniques of dealing large scales of logs from different kinds of equipment and constructing the hierarchical framework to fusion the pieces of information from the data layer to the feature level, and decision level.Keywords: cloud computing, data fusi

5、on, framework design1、引言 随着信息技术的不断发展和信息化建设的高速推进，网络已成为人们进行科研、生产、办公的重要平台。由于网络具有开放性、互联性、共享性的特点，其遭受入侵的风险也日趋严重，计算机网络安全问题日益突出。黑客活动日趋频繁，网站后门、网络钓鱼、恶意程序、拒绝服务攻击事件呈大幅增长态势，针对特定目标的有组织高级可持续攻击（APT攻击）日渐增多，涉密网络信息系统安全面临严峻挑战。为了保证计算机和网络的安全，军工单位的涉密网都部署了大量的网络安全设备（如防火墙、IDS）、主机监控系统、应用系统审计等用于增强网络安全防护和网络安全审计。与此同时，这些设备和系统产生了大

6、量的结构多样、彼此独立的日志信息，这些信息无法反映一次完整的攻击,只是记录了攻击的片段，因此如何将这些片断信息链接起来，重现整个网络攻击过程，发现攻击者的真正意图，是目前网络安全态势研究的重点和难点。本文基于以上需求，研究多源海量日志数据处理分析技术，构建层次型数据融合处理框架，实现从数据层到特征层，再到决策层的多源海量日志数据的融合处理，设计并实现基于云计算技术的网络告警数据分析系统。1 IntroductionWith the rapid advance of information technology and the continuous development of informat

7、ion technology, the network has become an important platform for scientific research, production, and office. As the network is open, interconnected, sharing features, the risk of being invaded is becoming serious and computer network security has become increasingly prominent. Hackers have become m

8、ore frequent, website backdoor, phishing, malicious programs, denial of service attacks showed a substantial growth, APT attacks become more frequent too.The security of secret network information system is facing serious challenges. In order to ensure the security of computer and network , military

9、 units network deploys a large number of network security devices (such as firewalls, IDS), host monitoring systems, applications, and auditing for enhanced network security and network security audits. At the same time, these devices and systems produce a large number of log information which is di

10、fferent in structure and independent of each other , which does not reflect a full attack, only recorded fragments of attack, so how these pieces of information linked to reproduce the entire network attack process, discover the true intent of the attacker, is currently the focus of network security

11、 situation and difficult research. Based on these requirements, researching the multi-source and massive data processing of log analysis techniques , constructing hierarchical and fusion data framework , achieving the characteristic layer from the data layer, and to he multi-source and massive data

12、processing of log analysis,designing and implementing cloud-based network Alarm data Analysis System computing technology are the tasks of this paper.2. 关键技术研究2.1 海量日志数据预处理技术首先需要对防火墙、IDS、主机监控系统的日志进行多源融合分析，日志数据进行集中采集，并保存在日志采集服务器上；然后再启动数据推送服务（可选择线上流量较小的凌晨）将日志数据文件推送至Hadoop平台；最后将日志文件写入到集群的HDFS中。对于防火墙、入侵

13、检测系统等设备来说，设备产生的日志信息可以通过syslog协议的方式进行推送，日志采集服务器端通过监听UDP或TCP端口的方式对日志进行采集。对于主机监控系统未提供syslog的方式推送日志，则需要读取主机监控系统服务器中的日志，并将其保存为文件形式待推送。日志数据生成过程如图1所示:2. Key Technology Research2.1 techniques of massive log data preprocessing Firstly the logs of firewall, IDS, host monitoring system for multi-source need to

14、 do fusion analysis, log data for centralized collected and stored in the log collection server; Then start data push service (optional wee small online traffic ) and files of logs data are pushed to Hadoop platform; finally the log files are written in HDFS cluster. For firewalls, intrusion detecti

15、on systems and other equipment, the log information generated by the equipment can be pushed through the way of the syslog protocol, log collection server use UDP or TCP port to listen tthe log collection. For syslog host monitoring system which does not provide a way to push the log, you need to re

16、ad the host surveillance system server log, and save it as a file and push to server. The process of log data generation shown in Figure 1:图1日志数据生成过程 Figure 1 The process of log data generation经过对防火墙、IDS、主机监控系统日志数据格式的研究，将日志分为四类（即管理配置异常类、流量异常类、违规操作类、安全攻击事件类）进行规范化处理。After the study of log data formats

17、 of firewall, IDS, and host monitoring system , the logs are divided into four categories (ie management configuration exception class, traffic anomaly class, illegal operations, security attacks classes) and normalized.（1）管理配置异常类管理配置异常类日志是指通过提取并分析防火墙、IDS、主机监控系统中管理配置相关的日志，发现异常的管理配置操作，通过对其日志的研究可将这类日志

18、规范化为如下格式：(1) Management Configuration exception classes Logs of Management Configuration exception classes are defined as logs extracted and analyzed from management configuration related logs of firewall, IDS,and host monitoring system,in order to find unusual configuration management operations ,

19、these logs can be normalized to the following format after studying :LogManager(ID,dev_type,event_type,priority,user,src_ip,op,time,result,msg)（2）流量异常类防火墙日志中连接类日志记录了每个session发送和接收的数据包大小，可通过统计该日志数据量来分析网络中异常流量。将这类日志规范化为如下格式：(2) traffic anomaly class The size of each session sended and received which i

20、s recorded in firewall logs can be used to analyse the anomaly of traffic .Such logs will be normalized to the following format:LogFlow(ID,dev_type,event_type,priority,src_ip,src_port,dst_ip,dst_port,time,proto,inpkt,outpkt,sent,rcvd)（3）违规操作类主要通过主机监控系统中产生的违规日志来分析违规操作，将这类日志规范化为如下格式：(3) illegal operat

21、ionsMainly through illegal logging host monitoring system to analyze the illegal operations, standardization of such logs like the following formats:Logillegal(ID,dev_type,event_type,user,pc_name,pc_ip,time,msg)（4）安全攻击事件类通过综合分析防火墙、IDS、主机监控系统三类日志发现潜在安全攻击事件，主要涉及到防火墙的访问控制类日志、IDS检测日志和主机监控中访问控制类日志，将这几类日志

22、规范化为如下格式：(4) security attacks classesA comprehensive analysis of firewall, IDS, host monitoring system logs can be used to find three categories of potential security attacks, which mainly related to the control class logs of firewall access control class logs, IDS logs and host monitoring , as foll

23、ows standardized format :LogSec(ID,dev_type,event_type,priority,src_ip,src_port,dst_ip,dst_port,time,proto)以上四类日志中各个属性表示的意义如表1所示：Meaning of each attribute as indicated in Table 1 below:表1各类日志属性与含义对应关系 Table 1 All kinds of correspondence between log attributes and meaning属性attributes 含义meaning备注Remar

24、kID日志IDLog IDdev_type设备类型Device Type如防火墙表示为fw，入侵检测系统表示为ids，主机监控系统表示为hm，再分别加上设备编号组成设备类型，如fw001、ids001、hm001。firewall is expressed as fw, intrusion detection system is expressed as ids, host monitoring system is expressed as hm, and then add the device number to consist device types, such as fw001, id

25、s001, hm001 respectively.event_type事件类型Event Type防火墙的管理配置类、访问控制类、连接类等，IDS自身定义的事件类型，主机监控系统的ftp、http、smtp、telnet、usb、print等类别。firewall configuration management class , access control class , connection class , etc., IDS itself defined event type, ftp, http, smtp, telnet, usb, print and other categorie

26、s of host monitoring system .priority事件优先级Event priority.分为8级：危急emergency(0)、报警alert(1)、严重critical(2)、错误error(3)、警告warning(4)、提示notice(5)、信息information(6)、调试debug(7)。Divided into eight: critical emergency (0), alarm alert (1), severely critical (2), error error (3), warning warning (4), prompt notic

27、e (5), information information (6), debug debug (7).user登录用户名Login Usernamesrc_ip源IPSource IPsrc_port源端口Source portdst_ip目的IPThe purpose of IPdst_port目的端口Destination portproto协议类型Protocol typeop操作的命令Operation Commandtime事件被记录的时间Time of events recordedresult事件结果Event results分为成功、失败。Into success, fail

28、ure.msg事件相关信息Event-related informationinpkt接收包数Number of received packetsoutpkt发送包数Send packetssent发送字节数Send bytesrcvd接收字节数Receive bytespc_name计算机名Computer Namepc_ip计算机ip地址Computer ip 本文采用HDFS文件系统来存储防火墙、IDS、主机监控系统的原始日志，并可设定日志采集服务器每探测到防火墙、IDS、主机监控系统中生成一条日志立即传输日志，这样避免黑客恶意删除原始日志。HDFS文件块存储示例如图2所示：In thi

29、s paper, HDFS file system is used to store raw logs of firewall, IDS, host monitoring system, and set the log collection server immediately transfer log when detecting firewall , IDS, host monitoring system generates a log , in order to avoiding malicious hacker deleted the original log . Example of

30、 HDFS file block storage shown in Figure 2: 图2HDFS文件块存储 图3管理配置类告警融合分析流程Figure 2 HDFS file block storage Figure 3 Alarm Management Configuration class fusion analysis process从图2中HDFS文件块存储示例中可以看出：主机监控系统原始日志（hm.log）备份数为3,分别存储于Datanode1、Datanode2、Datanode4三个节点上；防火墙原始日志（fw.log）备份数为2，分别存储于Datanode1、Datano

31、de3两个节点上；IDS原始日志（ids.log）备份数为2，分别存储于Datanode3、Datanode4两个节点上。这些文件存储的节点信息都在Namenode中有相应记录，当其中一个节点发生故障，Namenode会从另一个节点读取数据，从而避免单点故障导致的数据丢失或损坏问题。HDFS file blocks from the store can be seen in the example of Figure 2: backup number of raw logs of host Monitoring System (hm.log) is 3, are stored in Datan

32、ode1, the Datanode2, Datanode4 three nodes; backup number of firewall raw logs (fw.log) is 2, are stored in Datanode1, the Datanode3 two nodes; backup number of IDS raw logs (ids.log) is 2, are stored in Datanode3, the Datanode4 two nodes. These information files nodes are recorded in Namenode, when

33、 one node fails, Namenode will read data from another node, in oeder to avoid data loss or corruption after a single points failure .2.2网络告警融合分析方法与流程设计2.2 Methods and process design of network alarm fusion analysis 网络告警融合分析是指通过对防火墙、IDS、主机监控系统的海量多源日志数据，利用提出的规则策略，结合资产信息、脆弱性信息和关联知识库等信息进行综合分析，通过融合分析判断出网

34、络中真实发生的攻击事件。对应于四类规范化日志和制定的四类规则策略，本文将真实发生的告警分类四类，分别是：管理配置类告警、流量异常类告警、违规操作类告警、安全攻击类告警。以管理配置类告警为例，其融合分析方法及其流程设计如下：Network Alarm fusion analysis is to use the massive multi-source logs data of firewall, IDS, host monitoring system , with the help of the proposed rule strategy, combined with asset inform

35、ation, vulnerability information and other information associated with a comprehensive analysis of knowledge, through the integration of analysis to determine whether the real network attacks occur. Corresponding to the four types of logs designing four rules of standardization strategy , this artic

36、le divides the true alert into four categories, namely: configuration management class alarms, traffic anomaly class alarms, illegal operations class alarms, security attack class alarms . Like configuration management class alarms , the integration of analysis and process designed as follows: 通过管理配

37、置类规则来分析防火墙管理配置类规范化日志，可以有效分析出不在合法IP和用户范围内的管理配置类日志，并向用户产生告警信息，用于提醒用户存在非法IP和用户管理配置防火墙。管理配置类告警融合分析流程图如图3所示：Using configuration management class rules to analyze standardized logs of firewall configuration class , you can effectively analyze and manage the IP which is illegally and logs of users , and to

38、 generate alarm information for users, to alert users the presence of illegal IP and usersoperation of firewall configuration. Fusion analysis flow chart of Alarm Management Configuration class shown in Figure 3: HDFS中读取管理配置类日志文件流；Stream of reading configuration management class from HDFS按行读取日志文件流，根

39、据管理配置类规则对日志文件流进行匹配。若与规则完全匹配成功，且日志时间发生于正常工作时间，则说明该日志是正常行为，此条日志分析结束，判断日志数，日志数大于0，进入，否则进入；若与规则匹配不成功，进入步骤； reading the log file stream line by line, according to the configuration management class rules match log file stream . If exists an successful match with the rules , and the time of log occurs in

40、normal working hours, then the log is normal behavior, this is the end of this log analysis , determining the number of logs, the log is more than 0, enter , otherwise enter ; If the rule matches unsuccessfully, proceeds to step ;与资产库进行匹配。若匹配成功，则说明该日志是来自于内部人员对防火墙进行管理配置，属于内部越权管理行为，标志其风险级别为“中”，写入管理配置类

41、告警库中，此条日志分析结束，判断日志数，日志数大于0，进入，否则进入；若匹配不成功，则说明该日志来自于外部的管理配置，属于外部越权管理行为，标志其风险级别为“高”，写入管理配置类告警库中，此条日志分析结束，判断日志数，日志数大于0，进入，否则进入 match with the asset library. If the match is successful, then the log from internal staff who manage the firewall configuration, belong to an internal management acts , markin

42、g its risk level as medium, record in the configuration management alarm class database, this is the end of this entry log analysis, judging the number of logs, if the log number is more than 0, enter , otherwise enter ; If the match is unsuccessful, then the log management configuration is from the

43、 outside, belongs to the external ultra vires management behavior, marking its risk level as high, record in configuration management alarm class database, then the end of this entry log, ajudging the number of logs, the log is more than 0, enter , otherwise enter .管理配置类告警融合分析结束。Management Configura

44、tion class Alarm fusion analysis ends.3. 系统架构设计3 System Architecture Design.3.1总体架构及服务层次3.1 overall architecture and service levels基于Hadoop的网络告警融合分析系统通过日志采集服务器将网络中安全设备的日志数据采集并推送至Hadoop平台，再由Hadoop平台对日志进行安全存储、预处理、聚合、融合分析，最终将网络中的异常行为、攻击行为、违规行为分析出来发送至告警监测中心可视化展示，该系统具有如下特点：Hadoop-based network analysis s

45、ystem alarm using the log collection server push logs data of network security devices to the Hadoop platform, and then with the help of Hadoop platform store secure , pretreat ,aggregate fusion analysis logs , and lastly find the network abnormal behavior, aggressive behavior, illegal behavior and

46、sent to the alarm monitoring center visual display, the system has the following characteristics:随着网络中安全设备规模增加和产生的日志量的海量增长趋势，系统利用Hadoop平台搭建了私有云用于数据存储，保证原始日志数据安全可靠的存储。 With the increasing growth trend of network security device logs the size and quantity produced, the system uses Hadoop platform to b

47、uild a private cloud for data storage, raw log data can be safe and reliable stored.系统利用Hadoop平台对海量日志进行融合分析处理，有效提高了系统的分析处理效率。 the system uses Hadoop platform to analysis and process logs, effectively improve the efficiency of the system analysis and processing.系统搭建了告警监测中心，用户能够实时监测由日志分析得出的安全事件。 the s

48、ystem sets up alarm monitoring center, so the user can draw real-time monitoring of security events from log analysis.如图4所示为基于Hadoop的告警融合分析系统总体架构图，整个分布式告警融合分析系统分为日志采集端、日志分析处理中心、告警监测中心三部分。其中日志采集服务器主要负责采集网络中防火墙、入侵检测系统、主机监控系统的日志数据，并将日志推送至Hadoop平台中；日志分析处理中心主要负责对原始日志的安全存储、日志预处理、日志聚合、告警融合分析，产生真实攻击的告警信息，并发

49、送给告警监测中心；告警监测中心主要是供管理员直接查看当前异常、攻击、违规行为等事件。Figure 4 shows that the overall architecture diagram of alarm fusion analysis system based on Hadoop , the system is divided in 3 parts： the log collection terminal, log analysis and processing center. Log collection System is responsible for collecting the

50、log of the firewall, intrusion detection systems, host monitoring system and push logs to Hadoop platform; log analysis processing center is mainly responsible for the secure storage of raw logs, logs pretreatment, log aggregation , alarm fusion analysis, generate real attack alarms, and send them t

51、o alarm monitoring center; alarm monitoring center is mainly for administrators to directly view the current exception, attacks, violations and other events. 图4系统总体架构图 图5系统服务层次图Figure 4 Overall system architecture diagram Figure 5 Service-level system diagram在基于Hadoop的网络告警融合分析系统中，根据所提供的服务类型将整个系统划分为4

52、个层次，即应用层、适配层、云计算平台层和数据采集层。系统的服务层次如图5所示。Hadoop-based fusion analysis of network alarm system, which depends on the type of services provided by the entire system divided into four levels, namely the application layer, the adaptation layer, cloud computing platforms and data collection layer. Service-

53、level system is shown in Figure 5.（1） 数据采集层(1) data collection layer数据采集层是整个系统的数据来源，由采集服务器统一采集各安全设备的日志，然后集中启动数据推送服务，最后将日志文件写入到集群的HDFS中。Data collection layer is the data source of the whole system, because the log collection server unified collection of all safety equipment and services focused on st

54、arting of data push , and finally write the log file to the cluster in HDFS.（2） 云计算平台层(2) cloud computing platform layer以Hadoop作为支撑，是整个系统的数据处理中心，利用HDFS存储海量的日志数据，保证了数据的可靠存储和并行读写；利用MapReduce并行处理机制，为系统提供了强大的数据计算能力；云计算平台层提供了一系列应用接口，为整个系统提供高效稳定的数据持久化支持。 Hadoop as a support, is a data processing center of

55、 the whole system, the use of HDFS to store amounts of log data, ensuring reliable storage and parallel reading and writing; utilizing MapReduce parallel processing mechanism, the system provides a powerful data computing power; cloud computing platform provides a series of application interface lay

56、er, in order to improve efficiently and stable data persistence support for the entire system.（3） 适配层(3) Adaptation Layer在基于Hadoop的网络告警融合分析系统中，适配层位于应用层和服务器集群之间，为整个系统提供管理和服务，为应用层提供统一的标准化程序接口和协议。通信引擎负责整个系统的数据流和控制流的交互和传输；基础数据管理用于控制整个代理子节点以及分析对象配置。 Hadoop-based network alarm integration, analysis system

57、 adaptation layer which is between the application layer and server clusters, provides management and services for the entire system, provides standardized interface and protocol for the application layer. Interactive communication engine and transmission of the entire system is responsible for data flow and control of flow; basis data management is used for controlling the entire agent child node and object configuration.（4） 应用层(4) Application Layer应用层采用J2EE规范的JSP、HTML、SSH（struts2+spring+hibe