Apache Flume installation

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

Steps to Install

Step1.

Download apache flume from the link

http://ftp.heanet.ie/mirrors/www.apache.org/dist/flume/1.5.0.1/

Step2.

Extract the tar file

$sudo tar -xvf apache-flume-1.5.0.1-bin.tar.gz

Create a directory named /usr/lib/flume

$sudo mkdir -p /usr/lib/flume

Provide permissions for access

$sudo chown -R 777 /usr/lib/flume

Copy the extracted file to /usr/lib/flume

$sudo cp /home/hadoop1/apache-flume-1.5.0.1-bin /usr/lib/flume

Step2.

Set FLUME_HOME, FLUME_CONF_DIR environment variables in .bashrc file as shown below and add the flume bin directory to PATH environment variable

$sudo vim ~/.bashrc

Add the following at the end of .bashrc file

### Flume Home directory

export FLUME_HOME="/usr/lib/flume/apache-flume-1.5.0.1-bin"
export FLUME_CONF_DIR="$FLUME_HOME/conf"
export FLUME_CLASSPATH="$FLUME_CONF_DIR"

export PATH="$FLUME_HOME/bin:$PATH"

Step3.

In FLUME_CONF_DIR copy the flume-env.sh.temple and create a file flume-env.sh and provide JAVA_HOME path for environment variable

$sudo cp flume-env.sh.temple flume-env.sh $sudo vim flume-env.sh

Add the following at the end of flume-env.sh file

JAVA_HOME="usr/lib/java/jdk1.7.0_55"

Step4.

To verify whether flume is configured properly, just type the following command in the terminal

$ flume-ng --help