Learning

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application. Steps to Install Step1. Download apache flume from the link http://ftp.heanet.ie/mirrors/www.apache.org/dist/flume/1.5.0.1/ Step2. Extract the tar file $sudo tar -xvf apache-flume-1.5.0.1-bin.tar.gz Create a directory named /usr/lib/flume $sudo mkdir -p /usr/lib/flume Provide permissions for access $sudo chown -R 777 /usr/lib/flume Copy the extracted file to /usr/lib/flume $sudo cp /home/hadoop1/apache-flume-1.5.0.1-bin /usr/lib/flume Step2. Set FLUME_HOME, FLUME_CONF_DIR environment variables in .bashrc file as shown below and add the flume bin directory to

Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. While initially developed by Facebook, Apache Hive is now used and developed by other companies such as Netflix Amazon maintains a software fork of Apache Hive that is included in Amazon Elastic MapReduce on Amazon Web Services Steps for installation 1. First you need to download the apache hive you can download the latest mirror of Apache Hive from here http://www.eu.apache.org/dist/hive/hive-1.2.1/apache-hive-1.2.1-bin.tar.gz 2.Untar the package which id downloaded using the following command: sudo tar –xvzf apache-hive-1.2.1-bin.tar.gz 3. Move the extracted package to hive folder sudo mv apache-hive-1.2.1-bin hive 4. Provide the appropriate permission for the hive folder sudo chown –R hduser:hdgroup hive 5. Next move the folder to /usr/local sudo mv hive /usr/local/ 6.Now edit the ~/.bashrc file and add the following contents a