尚大数据技术之flume

1、(作者：大海哥3.1Flume 2）Flume (作者：大海哥3.1Flume 2）Flume 常3.2 Flume （1）单个（2）多级agent （1）（1）单个（2）多级agent （1）Sink可以根据SinkGroup和sor由Sink。（3）) 三3.1 Flume 的安装 下2）解压apache-flume-1.7.0-) 三3.1 Flume 的安装 下2）解压apache-flume-1.7.0-下 3）修改apache-flume-1.7.0-bin名称为 doop102flume#viexport export doop102flume#source Flumenc 源到

2、控1）需求：从Linuxnc： doop102module#yuminstalldoop102module#nc-，并修改名称为 （2/opt/module/flume/conf/perties.template， 并修改名称为 doop102module#nc-，并修改名称为 （2/opt/module/flume/conf/perties.template， 并修改名称为 export （c）描述配置ink到在下创建配置文件flume-= r a.sinks = k a.sour .r.type = netcat a.sour .r.

3、bind= a.sour .r.port = 8888a.sinks.k.type=# Use a channel which buffers eventsa.channels.c.type = memory a.channels.c.capacity = 1000#Bindthesourceandsinktothechannel .r.channels = c a.sinks.k.channel = c#-conf - #-conf - 3.2 小试先在flume的# 定义这个agent= r1 a1.sinks = k1 #描述和配置source组件：r1 .r1.type

4、= netcat（2）启动agent-c指定flume-f-n 指这个agent的名 （2）启动agent-c指定flume-f-n 指这个agent的名 （ netlocalhost到HDFS （2）下沉目标，即sinkHDFS文件系统 bin/flume-ngagent-cconf-fogger.conf-n-.r1.port = 44444sink组件：k1 a1.sinks.k1.type = logger# a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 ionCapacity=100# 描述和配置 .r1.ch

5、annels=c1 a1.sinks.k1.channel = c1（1）先在flume（1）先在flume的event = r1 a1.sinks = k1 a1.channels = c1# source.r1.type = spooldir .r1.fileHeader = true# sinka1.sinks.k1.type= a1.sinks.k1.hdfs.filePrefix = aca1.sinks.k1.hdfs.maxOpenFiles=a1.sinks.k1.hdfs.batchSize= 100 a1.sinks.k1.hdfs.writeFormat =Text a1

6、.sinks.k1.hdfs.rollSize = 102400a1.sinks.k1.hdfs.rollCount = 1000000 erval = 60 # 配置 channels 组件 a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 # .r1.channels=c1 a1.sinks.k1.channel = c1 ： ： tail-F （2）下沉目标，即sinkHDFS文件系统 （3）Source 和 sink 之间的传递通道channelfile channel = r1 a1.sinks = k1 a1.c

7、hannels = c1# source.r1.type=mand=tail-Fbin/flume-ngagent-c./conf-f./conf/spool-logger.conf-na1- atguiguhadoop102 atguiguhadoop102logs$tail-Fatguiguhadoop102datas$whileechoatguigusleepbin/flume-ngagent-cconf-fconf/tail-hdfs.conf-n.r1.channels=# 配置 sink 组件 a1.sinks.k1.type = hdfs a1.sinks.k1.hdfs.fil

8、ePrefix = events-a1.sinks.k1.hdfs.round = true a1.sinks.k1.hdfs.roundValue = 10 erval = 3 a1.sinks.k1.hdfs.rollSize = 20 a1.sinks.k1.hdfs.batchSize = 1 # 配置 channels 组件 a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 ionCapacity=100# .r1.channels=c1 a1.sinks.k1.channel = c13.5 agent3.5 a

9、gent a）在flume#Namethecomponentsonthisagent = r1a1.sinks = k1 .r1.type = execmand=tail-F/opt/logs/test.log .r1.channels = c1# Describe the sink a1.sinks = k1a1.sinks.k1.type =avro a1.sinks.k1.channel=c1 a）在flume #Namethecomponentsonthisagent = r1a1.sinks = k1 .r1.type = avro .r1.channels = c1 .r1.bin

10、d = .r1.port = 4141# Describe the sink # Use a channel which buffers eventsa1.channels.c1.type = memory a1.channels.c1.capacity = 1000 ionCapacity = 100#Bindthe sourceandsinktotheatguiguhadoop102datas$whileechoatguigusleep a1.sinks.k1.port = 4141 # Use a channel which buffers eventsa1.channels.c1.type = memory a1.channels.c1.capacity = 1000 ionCapacity = 100#Bindthesourceandsinktothechannel .r1.channel