Apache Spark

 Spark on Windows

==========================================

Step 1: Download and  install JDK

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html


Step 2: Download Spark

http://spark.apache.org/downloads.html

Choose the following options for the download
1. Choose a Spark release - 1.6.1
2. Pre-built for Hadoop 2.6 or later
3. Direct Download
4. Click on the spark-1.6.1-bin-hadoop2.6.tgz


Step 3: Extract the tar file

Step 4: Copy the contents of the tar files into 

C:\spark\ folder

Step 5: Update the log4j.properties to set the messages to WARN

C:\spark\conf\log4j.properties.template

Set the property - log4j.rootCategory=WARN

Save the file as log4j.properties

Step 6: Download winutils.exe from the here

1. Create a folder C:\winutils\bin
2. Copy the winutils.exe file into C:\bin\winutils.exe

Step 7: Set the environment variables (Inform Windows where is Spark)


SPARK_HOME =  C:\spark
JAVA_HOME = C:\Program Files\Java\jdk1.8.0_71
HADOOP_HOME = C:\winutils
PATH = %SPARK_HOME%\bin;%JAVA_HOME%\bin (append to the existing path)

In Windows
==========

1) Right click on Start Menu
2) Click on Control Panel
3) Click on System and Security
4) Click on System
5) Click on Advanced System Settings
6) Click on Environment Variable button
1. Add New Environment Variable
2. Variable Name: SPARK_HOME, Variable Value: C:\spark, Save the variable
3. Variable Name: JAVA_HOME, Variable Value: C:\Program Files\Java\jdk1.8.0_71
4. Variable Name: PATH, Variable Value (append to the end of the string) %SPARK_HOME%\bin;%JAVA_HOME%\bin
5. Variable Name: HADOOP_HOME, Variable Value: C:\winutils


Step 8: Open a terminal and start the spark shell

1. Command Prompt App
2. cd C:\spark\bin
       3. spark-shell