Hadoop on Windows with Eclipse

1. Introduction
2. Prerequistes
3. Set Environment Variables
4. Setup SSH daemon
5. Download hadoop and place it in the home directory.
6. Unpack hadoop
7. Configure Hadoop
8. Format the namenode
9. Setup hadoop plugin
10. Start the cluster
11. Setup hadoop location
12. Upload data
13. Create and run a test project.

Create and run Hadoop project

Now we are ready to create and run out first Hadoop project.

Creating and configuring Hadoop eclipse project.

1. Launch Eclipse
2. Right click on the blank space in the Project Explorer window and select New -> Project.. to create a new project.
3. Select Map/Reduce Project from the list of project types. As shown on the image below.
   
   
4. Press the Next button.
5. You will see the project properties window similar to the one shown below
   
   
6. Fill in the project name and then click on Configure Hadoop Installation link. Which is located on the right side of the project configuration window. This will bring up the Project preferences window shown on the image below.
   
   
   
   
7. When Project preferences window shows up, enter the location of the hadoop directory in the Hadoop Installation Directory field as shown above.
   
   If you are not sure what is your Hadoop home directory location. Follow the refer to the step 1 of this section. The hadoop home directory is one level up from the conf directory.
   
   
8. After you entered the location close the preferences window by pressing OK button, and then close the Project window by the Finish button.
9. Now you have created your first Hadoop eclipse project. You should see its name in the Project Explorer tab.

Creating Map/Reduce driver class

1. Right click on the newly created Hadoop project in the Project Explorer tab and select New -> Other from the context menu.
2. Go to Map/Reduce folder, select MapReduceDriver then press the Next button. As shown on the image below.
   
   
3. When MapReduce Driver wizard appears enter the TestDriver in the name field and press the Finish button. This will create the skeleton code for the MapReduce Driver.
   
   
4. Unfortunately the Hadoop plugin for eclipse is slightly out of step with the recent Hadoop API, so we need to edit the driver code a bit.
   
   Find two following lines in the source code and comment them out:
   
   conf.setInputPath(new Path("src"));
   conf.setOutputPath(new Path("out"));

   Enter the following code right immediatly below the two lines you just commented out.

   conf.setInputFormat(TextInputFormat.class);
   conf.setOutputFormat(TextOutputFormat.class);
   
   FileInputFormat.setInputPaths(conf, new Path("In"));
   FileOutputFormat.setOutputPath(conf, new Path("Out"));
   
   
   As shown on the image below
   
   
5. After you have changed the code, you will see the new lines marked as incorrect by the Eclipse. Click on the the error icon for each of the line and select Eclipse's suggestion to import the missing class.
   
   You need to import the following classes TextInputFormat, TextOutputFormat, FileInputFormat, FileOutputFormat.
   
6. After the missing classes are imported we are ready to run our project.

Running Hadoop Project

1. Right click on the TestDriver class in the Project Explorer tab and select Run As --> Run on Hadoop. This will bring up the windows like the one shown below.
   
   
2. In the window shown above select "Choose existing hadoop location" , then select localhost from the list below. After that click Finish button to start your project.
3. If you see console output similar to the one shown below, congratulations you have started the project successfully.
   
   

Continue