Hadoop on Windows with Eclipse

1. Introduction
2. Prerequistes
3. Set Environment Variables
4. Setup SSH daemon
5. Download hadoop and place it in the home directory.
6. Unpack hadoop
7. Configure Hadoop
8. Format the namenode
9. Setup hadoop plugin
10. Start the cluster
11. Setup hadoop location
12. Upload data
13. Create and run a test project.

Upload data to HDFS

Now we are almost ready to run our first Map/Reduce project. The only thing is missing is the data. This section will explain how to upload the data into Hadoop Distributed File System.

Upload Files To HDFS

1. Open a new CYGWIN command window.

   

2. Execute the following commands in the new CYGWIN window as shown on the image above.
   

   cd hadoop-0.19.1
   bin/hadoop fs -mkdir In
   bin/hadoop fs -put *.txt In

   When the last of the above commands will start execution you should see some activity happening in the rest of the hadoop windows as shown on the image below.

   
   

   The result of these commans is a newly created directory in the HDFS named In which contains a set of text files that comes with the hadoop distribution.

3. Close the Cygwin Window.

Verify if the files were uploaded correctly

In the section we will check if the files were uploaded correctly.

1. Open Eclipse Environment

2. Open DFS locations folder which is located in Project Explorer tab of Map/Reduce perspective.
3. Open localhost folder which is contain in the DFS locations folder
4. Keep opening HDFS folders until you navigate to the newly created In directory. As shown on the image below.
   

   

   Verifying that the data was uploaded correctly

   
   

5. When you get to the In directory double-click on the file LICENCE.TXT to open it.
6. If you see something similar to the image above then the data was uploaded correctly and we can proceed to creating our first Hadoop project.

Continue