认证数据工程师Flume实验

Flume实验1.TDH4.8以前（包括TDH4.8）使用yum方式从本地源安装flume#yuminstallflumeTDH4.8.1开始通过flume官方网站下载flume安装包，放到/mnt/disk1下。2.配置环境变量修改/etc/profile，输入：vim/etc/profile加上参数:exportJAVA_HOME=/usr/java/jdk1.7.0_71exportHADOOP_HOME=/usr/lib/hadoopexportHDFS_HOME=/usr/lib/hadoop-hdfsexportHADOOP_CLASSPATH=$HADOOP_HOME:$HADOOP_HOME/lib:$HDFS_HOME/lib3.创建flume-env.sh文件。进入flume目录：cd/$FLUME_HOME/flume/conf创建：vimflume-env.sh内容如下：FLUME_HOME="/usr/lib/flume"FLUME_CLASSPATH="$FLUME_HOME/lib"JAVA_HOME="/usr/java/jdk1.7.0_71"JAVA_OPTS="-Xms512m-Xmx1024m-Dcom.sun.management.jmxremote"HADOOP_HOME="/usr/lib/haoop"HDFS_HOME="/usr/lib/hdfs"HADOOP_CLASSPATH="$HADOOP_HOME:$HADOOP_HOME/lib:$HDFS_HOME/lib"4.在/$FLUME_HOME/conf目录下编辑flume.conf配置文件，设置source类型为分别为目录、文件，sink类型为hdfsflume.conf内容：#a.conf:Asingle-nodeFlumeconfiguration#Namethecomponentsonthisagenta1.sources=r1a1.sinks=k1a1.channels=c1#Describe/configurethesourcea1.sources.r1.type=spooldira1.sources.r1.spoolDir=/opt/vdb/loga1.sources.r1.fileHeader=truea1.sources.r1.deserializer.outputCharset=UTF-8#Describethesinka1.sinks.k1.type=hdfsa1.sinks.k1.hdfs.path=hdfs://tdh-101:8020/tmp/bxu/file1a1.sinks.k1.hdfs.fileType=DataStreama1.sinks.k1.hdfs.writeFormat=Texta1.sinks.k1.hdfs.maxOpenFiles=1a1.sinks.k1.hdfs.rollCount=0a1.sinks.k1.hdfs.rollInterval=0a1.sinks.k1.hdfs.rollSize=1000000a1.sinks.k1.hdfs.batchSize=10000#Useachannelwhichbufferseventsinmemorya1.channels.c1.type=memorya1.channels.c1.capacity=1000000a1.channels.c1.transactionCapacity=10000#Bindthesourceandsinktothechannela1.sources.r1.channels=c1a1.sinks.k1.channel=c15.在本地创建flume监控的文件目录或者文件，并设置该目录或者文件的权限为777操作命令记录：如上图所示的demo，上图中的红色部分目录需要手动创建。#mkdir-r/opt/vdb/log/#chmod777-R/opt/vdb6.执行以下命令打开flume的常驻进程，对指定的文件目录或者文件进行监控#cd/usr/lib/flume#bin/flume-ngagent--confconf--conf-fileconf/flume.conf--namea1-Dflume.root.logger=INFO,console以上内容为，监控文件夹，如果linux目录的文件夹下，有文件添加，那么会自动采集到HDFS目录，如果需要监控具体的文件内容，如果该文件中有数据更新，那么需要修改flume.conf文件为如下，其他不变！#a.conf:Asingle-nodeFlumeconfiguration#Namethecomponentsonthisagenta1.sources=r1a1.sinks=k1a1.channels=c1#Describe/configurethesourcea1.sources.r1.type=execmand=tail-F/opt/vdb/log/exec.texta1.sources.r1.fileHeader=truea1.sources.r1.deserializer.outputCharset=UTF-8#Describethesinka1.sinks.k1.type=hdfsa1.sinks.k1.hdfs.path=hdfs://tdh-101:8020/tmp/bxu/file1a1.sinks.k1.hdfs.fileType=DataStreama1.sinks.k1.hdfs.writeFormat=Texta1.sinks.k1.hdfs.maxOpenFiles=1a1.sinks.k1.hdfs.rollCount=0a1.sinks.k1.hdfs.rollInterval=0a1.sinks.k1.hdfs.rollSize=1000000a1.sinks.k1.hdfs.batchSize=10000#Useachannelwhichbufferseventsinmemorya1.channels.c1.type=memorya1.channels.c1.capacity=1000000a1.channels.c1.transactionCapacity=10000#Bindthesourceandsinktothechannela1.sources.r1.cha