flume安装配置

flume

参考： Flume NG 简介及配置实战 Flume介绍与安装

1、flume 下载

    flume下载的地址

2、安装

  将安装包放到安装路径， 解压:

//解压
tar -zxf apache-flume-1.7.0-bin.tar.gz
//移到指定路径 
sudo mv apache-flume-1.7.0-bin /usr/local/flume-1.7.0

3、配置参数 3.1、 配置/etc/profile 参数

编辑/etc/profile文件，声明flume的home路径和在path加入bin的路径：

export FLUME_HOME=/usr/local/flume-1.7.0
export FLUME_CONF_DIR=$FLUME_HOME/conf
export PATH=$PATH:$FLUME_HOME/bin

编译配置文件/etc/profile，并确认生效

source /etc/profile
echo $PATH

3.2、设置flume-env.sh配置文件

在$FLUME_HOME/conf 下复制改名flume-env.sh.template为flume-env.sh，修改conf/ flume-env.sh配置文件:

JAVA_HOME= /app/lib/jdk1.7.0_79
JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"

3.3 验证安装

3.3.1、修改flume-conf配置文件 在$FLUME_HOME/conf目录下修改flume-conf.properties.template文件，复制并改名为 flume-conf.properties， 编辑flume-conf.properties：

/usr/local/flume-1.7.0/conf
cp flume-conf.properties.template flume-conf2.properties

3.3.2、修改flume-conf.properties：

# The configuration file needs to define the sources, the channels and the sinks.
# Sources, channels and sinks are defined per agent, in this case called 'a1'
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# For each one of the sources, the type is defined
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

#The channel can be defined as follows.
a1.sources.r1.channels = c1
# Each sink's type must be defined
a1.sinks.k1.type = logger

#Specify the channel the sink should use
a1.sinks.k1.channel = c1

# Each channel's type is defined.
a1.channels.c1.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

3.3.3、在flume的安装目录/flume-1.7.0下运行

cd /usr/local/flume-1.7.0
./bin/flume-ng agent --conf ./conf/ --conf-file - ./conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console

PS：-Dflume.root.logger=INFO,console 仅为 debug 使用，请勿生产环境生搬硬套，否则大量的日志会返回到终端。。。

-c/–conf 后跟配置目录，-f/–conf-file 后跟具体的配置文件，-n/–name 指定agent的名称

出错：

2016-11-20 08:54:34,702 (main) [ERROR - org.apache.flume.node.Application.main(Application.java:348)] A fatal error occurred while running. Exception follows.
org.apache.commons.cli.ParseException: The specified configuration file does not exist: /usr/local/flume-1.7.0/conf/flume-conf2.properties
        at org.apache.flume.node.Application.main(Application.java:316)

解决：

由于上面的 flume-conf2.properties修改成了 flume-conf.properties， 改一下文件名

3.4、 测试收集日志到HDFS

修改 flume-conf2.properties：

a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type = exec
a1.sources.r1.channels = c1
//监视的日志文件
a1.sources.r1.command = tail -F /usr/local/hadoop/logs/hadoop-chb-namenode-TEST.log
a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
//输出到hdfs
a1.sinks.k1.hdfs.path = hdfs://192.168.1.124:9000/output/out_flume
//输出文件前缀
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.rollSize = 4000000
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.batchSize = 10
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

运行flume

cd   /usr/local/flume-1.7.0
//运行flume
./bin/flume-ng agent --conf ./conf/ --conf-file ./conf/flume-conf2.properties --name a1 -Dflume.root.logger=INFO,console

查看hdfs中 /output/out_flume中的文件：

hadoop fs -cat  /output/out_flume/
hadoop fs -cat  /output/out_flume/events-.1479642835192