10、flume应用场景原理基本架构与案例分析

1、Flume OG基本架Flume OG基本架Flume NG基本架Flume案例分6Flume OG基Flume OG基本架7Flume OG基Flume OG基本架8通常由source和部V通常由source和部V组Source用于获取数据，从文本文件，syslog，HTTPSink将Source获得的数据进一步传输给面的Flume自带了很多source和sinksyslogTcp(5140) |)|9collectorSource(35853) | console collectorSource(35853) | collectorSink(file:/tmp/flume/collecte

2、d,syslog); collectorSource(35853) |collectorSink(hdfs:/namenode/user/flume/ AgentPCollector对应AgentPCollector对应关AgentPCollector对应关AgentPCollector对应关2Flume集群的制器借zookeeper管理管理多容错机制容错机制容错机制容错机制Flume OG基本架Flume Flume OG基本架Flume NG基本架Flume案例分Flume NG基Flume NG基本架Flume NG心概CFlume NG心概C其他组件erceto1Cannl lo1iE

3、vent是Flume数据传输的基Event是Flume数据传输的基本单e件的形式将数传送到最Event由选的header和载有数据的一个bytearray成载有的数据对flume是O透 e在PQ文路由中使用扩展Cnt是一个将原始log包装成events个agent的实体Cnt是一个将原始log包装成events个agent的实体在flume的拓扑结构中O是必ntFlume log4j C使用.apache.flume.api)定制特定ntSDKC一个Agent包含Source,一个Agent包含Source, Channel, Sink和其他组件agent是flume流的基础部Agent之Ag

4、ent之Agent之Source负责接收event或通过特殊机制产生eventAgent之Source负责接收event或通过特殊机制产生eventO类型的P系统集成的Source: Syslog, 自生成件的Source: 用于Agent和Agent之间通信的IPC Source: Source必至少和一个channel关Agent之PEvent Agent之PEvent Event Agent之Agent之O的Channel提供的持久化水平也是O一的MemoryChannel: FileChannel基于WAL预写式日志Write-Ahead JDBC Channel: 基于嵌入Datab

5、aseChannel支持，提供较弱的和任何数量的Source和Sink工Agent之Sink负责将event传输到QAgent之Sink负责将event传输到Q一跳或最终目的，成成将event从channel移除O类型的储event到最终目的的终端Sink. 如: HDFS, 自消耗的Sink. 如: Null 用于Agent间通信的IPC sink: 必作用于一个确W的其他几个作用于Source，按照预设的其他几个作用于Source，按照预设的序在必要地方装饰和过滤Channel允许Source基于预设的标准，从所有Channel中，选择一个或alSink多个一个sor通过组中所有实现负载均

6、衡也在一个Sink失败时转移到另一个Flume OG基本架Flume OG基本架Flume NG基本架Flume案例分Q步骤2修改etc/profileexport FLUME_HOME=/opt/software/flume-0.9.4- export步骤3验证安步骤5运行命启OG集群态OG集群态在已经启的master节点P，依次输入localhost s如执行exec config a1 tailDir(/data/logfile lume.collector.port两个参数决定和点，填写source1sink选OG集群拓扑配置OG集群拓扑配置agentA:src|agentB : sr

7、c | agentSink(collector,35853); agentC : src | agentSink(collector,35853); agentD :src | agentSink(collector,35853); agentE : src | agentSink(collector,35853); agentF : src | agentSink(collector,35853); collector : collectorSource(35853) |OGO种靠性级end to end，这个级OGO种靠性级end to end，这个级别是sonanacknowledgem

8、ent, andretry if no acknowledgement is 效率最好，agentO写入到本地任何数据，如果在直接删除消息发现处理失败，AgentSink 是agentE2ESink 的别OG集群拓扑配置OG集群拓扑配置agentA:src|agentB : src | agentE2ESink(collectorA,35853); agentC : src | agentE2ESink(collectorB,35853); agentD : src |agentE2ESink(collectorB,35853); agentE : src | agentE2ESink(col

9、lectorC,35853); agentF : src | agentE2ESink(collectorC,35853);collectorA :collectorSource(35853) | collectorSink(hdfs:/.,src); collectorB : collectorSource(35853) | collectorSink(hdfs:/.,src); collectorC : collectorSource(35853) | collectorSink(hdfs:/.,src);OG集群拓扑配置OG集群拓扑配置agentA:src|agentB : src |

10、agentE2EChain(collectorA:35853,collectorC:35853); agentC : src | agentE2EChain(collectorB:35853,collectorA:35853); agentD : src |agentE2EChain(collectorB:35853,collectorC:35853); agentE : src | agentE2EChain(collectorC:35853,collectorA:35853); agentF : src | agentE2EChain(collectorC:35853,collectorB

11、:35853); collectorA : collectorSource(35853) | collectorSink(hdfs:/.,src); collectorB : collectorSource(35853) | collectorSink(hdfs:/.,src); collectorC : collectorSource(35853) | collectorSink(hdfs:/.,src);Q步骤2修改etc/profileexport FLUME_HOME=/opt/software/flume-1.4.0.tar.gz export PATH=.:$PATH:$FLUME

12、_HOME/bin步骤3验证安步骤5运行命启NG部署-配置文件格Java PropertiesNG部署-配置文件格Java Properties 文件格# 注释 key1valuekey2=multi-line层级配agent1.channels.myChannel.type = = 使用配关.mySource.type= .mySource.channels=NG部署-简单配置实# 定OsourceNG部署-简单配置实# 定Osource1channel和sink=agent1.channels = agent1.sinks=.src1.type= .src1.channels=.src1.

13、bind=.src1.port = #定O并配置 sink1 agent1.sinks.sink1.type = logger agent1.sinks.sink1.channelch1# 定O并配置 ch1 agent1.channels.ch1.typememoryNG部署-实例NG部署-实例NG部署-实例第一层1配置实例a1.channels=a1.sinks.k1.channelNG部署-实例第一层1配置实例a1.channels=a1.sinks.k1.channel=a1.sinks.k1.type = AVRO a1.sinks.k1.hostname=a21.ex a1.sin

14、ks.k1.port = 41414=a1.sinks=k1sor.type=sor.backoff = truea1.sinks.k2.channel=a1.sinks.k2.type = AVRO a1.sinks.k2.hostname=a22.ex a1.sinks.k2.port = 41414.r1.channels= .r1.type=.r1.port=NG部署-实例第层2配置实例NG部署-实例第层2配置实例a2.channels = =a2.sinks = a2.channels.c1.type = .r1.channels=.r1.type=.r1.bind=.r1.port

15、 = a2.sinks.k1.channel = a2.sinks.k1.type = HDFS a2.sinks.k1.hdfs.path = hdfs:/namenode.ex a2.sinks.k1.hdfs.fileType=DataStreamNG部署-实例NG部署-实例NG部署-实例a1.channels=a1.sinks=NG部署-实例a1.channels=a1.sinks=k1a1.sinkgroups = g1 a1.sinkgroups.g1.sinks=k1k2sor.type=sor.backoff = truea1.channels.c1.type=第一层实1配.r

16、1.channels=.r1.host=a1.sinks.k1.channel=a1.sinks.k1.type = AVRO a1.sinks.k1.hostname=a21.ex a1.sinks.k1.port = 41414 a1.sinks.k2.channel = c1 a1.sinks.k2.type = AVRO a1.sinks.k2.hostname=a22.ex a1.sinks.k2.port = 41414NG部署-实例第层2配置实例.r1.channels=c1.r1.type = NG部署-实例第层2配置实例.r1.channels=c1.r1.type = .r

17、1.bind= .r1.port =.r1.selector.header= .r1.selector.default= =a2.sinks = k1 k2 a2.sinkgroups.g1.sinks=k1sor.type=.0=c1 .1=c1 .2=c1 .3=c1 sor.backoff = true Dir=/var/run/flume-ng/flume/ch-1/checkpo a2.channels.c1.dataDirs = /var/run/flume-ng/ flume/ch-1/dataa2.sinks.k1.channel=a2.sinks.k1.type = HDFS a2.sinks.k1.hdfs.path = hdfs:/nn.exa2.channels.c2.type=Dir=/var/run/flume-ng/flume/ch-2/checkpo a2.channels.c2.dataDirs = /var/run/flume-ng/ flume/ch-2/data/demo/%Y-a2.sinks.k1.hfds.filePrefix = FlumeData-a2.sinks.k1.hdfs.fileType=DataStream a2.sin