Apache Sqoop

INTRODUCTION

Sqoop is a tool designed to transfer data between Hadoop and relational databases or mainframes. You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle or a mainframe into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS.

Sqoop automates most of this process, relying on the database to describe the schema for the data to be imported. Sqoop uses MapReduce to import and export the data, which provides parallel operation as well as fault tolerance.

Steps for Installation

Step 1 - Download sqoop-1.4.4.bin hadoop-1.0.0.tar.gz from the mirror website sqoop.apache.org

wget http://mirror.cogentco.com/pub/apache/sqoop/1.4.4/sqoop-1.4.4.bin__hadoop-1.0.0.tar.gz

Step 2 - Untar the downloaded file

tar -xvzf sqoop-1.4.4.bin__hadoop-1.0.0.tar.gz

Step 3 - Copy the extracted folder in /usr/local/sqoop location

sudocp -r sqoop-1.4.4.bin__hadoop-1.0.0.tar.gz /usr/local/sqoop

Step 4 - Set the path in bash environment

sudo vi $HOME/.bashrc

type the following at the end of the line

export PATH=$PATH:/usr/local/sqoop/bin

export HADOOP_COMMON_HOME=/usr/local/hadoop

export HADOOP_MAPRED_HOME=/usr/local/hadoop

export HIVE_HOME=/usr/local/hive

export HBASE_HOME=/usr/local/hbase

Step 5 - To perform practical on SQOOP, ensure you have mysql server installed in the server machine. The command to install mysql is

sudo apt-get install mysql-server

Step 6 - Start mysql server using command

mysql -u root -p

Step 7 - Create a database for testing sqoop

create database imran;

Step 8 - Create a table named auth inside imran database.

use imran;

create table auth(username varchar(30),password varchar(30));

Step 9 - Add some entries in table.

insert into auth values ('admin','123456');

insert into auth values ('imran ','imran123');

Step 10 - Download mysqljdbc connector. Extract the zip file and copy the jar file in /usr/local/sqoop/lib.

Step 11 - Add permissions

Sudo chown username /usr/local/sqoop

Sudo chown username /usr/local/sqoop/lib

Step 12 - To check if the sqoop has been installed successfully type the command

sqoop version