By | August 10, 2015

Hadoop installation on windows without cygwin in 10 mints – Hadoop installation on windows 7 or 8

Download
Before starting make sure you have this two softwares

Extract downloaded tar file

Configuration
Step 1 – Windows path configuration
set HADOOP_HOME path in enviornment variable for windows
Right click on my computer > properties > advanced system settings > advance tab > environment variables > click on new

Set hadoop bin directory path
Find path variable in system variable > click on edit > at the end insert ‘; (semicolon)’ and paste path upto hadoop bin directory in my case it’s a

F:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/bin


Step 2 – Hadoop configuration
Edit hadoop-2.7.1/etc/hadoop/core-site.xml, paste the following lines and save it.

<configuration>
<property>
       <name>fs.defaultFS</name>
       <value>hdfs://localhost:9000</value>
   </property>
</configuration>

Edit hadoop-2.7.1/etc/hadoop/mapred-site.xml, paste the following lines and save it.

<configuration>
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
</configuration>

Edit hadoop-2.7.1/etc/hadoop/hdfs-site.xml, paste the following lines and save it, please create data folder somewhere and in my case i have created it in my HADOOP_HOME directory

<configuration>
<property>
       <name>dfs.replication</name>
       <value>1</value>
   </property>
   <property>
       <name>dfs.namenode.name.dir</name>
       <value>/f:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/data/namenode</value>
   </property>
   <property>
       <name>dfs.datanode.data.dir</name>
     <value>/f:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/data/datanode</value>
   </property>
</configuration>

Edit hadoop-2.7.1/etc/hadoop/yarn-site.xml, paste the following lines and save it.

<configuration>
   <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
   </property>
   <property>
       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
       <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>
</configuration>

Edit hadoop-2.7.1/etc/hadoop/hadoop-env.cmd, comment existing %JAVA_HOME% using @rem at start, give proper path and save it. (my jdk is in program files to avoid spaces i gave PROGRA~1)



Demo

Step 3 – Start everything

Very Important step

Before starting everything you need to add some [dot].dll and [dot].exe files of windows please download bin folder from my github repository – sardetushar_gitrepo_download

bin folder – this contains .dll and .exe file (winutils.exe for hadoop 2.7.1)

Now delete you existing bin folder and replace with new one (downloaded from my repo )

( my github ) etc folder is just given for reference you need to modify your configuration parameters according to your environment path

Open cmd and type ‘hdfs namenode -format’ – after execution you will see below logs

Open cmd and point to sbin directory and type ‘start-all.cmd

F:Hortanwork1gbhadoopramSoftwarehadoop-2.7hadoop-2.7.1sbin>start-all.cmd

It will start following process

Namenode

Datanode

YARN resourcemanager

YARN nodemanager


JPS – to see services are running
open cmd and type – jps (for jps make sure your java path is set properly)

GUI
Step 4 – namenode GUI, resourcemanager GUI
Resourcemanager GUI address – http://localhost:8088

Namenode GUI address – http://localhost:50070

In next tutorial we will see how to run mapreduce programs in windows using eclipse and this hadoop setup




Share this knowledge ! Join us on Facebook ! Now Whatsapp sharing is supportable ! TooDey Inc.

42 thoughts on “

  1. akash yadav

    Very much helpful and easy process. but when i run this Command”hdfs namenode -format” i got an error
    like this:>
    c:\>hdfs namenode -format (Enter)
    c:\>could not find or load main class yadav

    Reply
  2. jkhan

    Thanks. I worked for me.
    Where can i read next tutorial?

    Reply
  3. kumar

    Hi Tushar,

    Thanks for the detail information. After all steps performed and when I provide start-all.cmd I get the following error
    The script is Deprecated instead use start-dfs.cmd and start-yarn.cmd. And when start-dfs.cmd is executed I get error as “The system cannot find the file hadoop”.
    Please help me on this. Thanks

    Reply
  4. Shalini Nigam

    Hi,

    I downloaded sucessfully as per steps given above.

    However while runing through cmd it is giving me below error
    C:\hadoop-2.7.1>hdfs namenode -format
    ‘Files’ is not recognized as an internal or external command,
    operable program or batch file.
    Error: JAVA_HOME is incorrectly set.
    Please update C:\hadoop-2.7.1\conf\hadoop-env.cmd
    ‘-Dhadoop.security.logger’ is not recognized as an internal or external command,

    operable program or batch file.

    C:\hadoop-2.7.1>

    Though I set JAVA_HOME correctly. Also I am not able to see conf folder under C:\hadoop-2.7.1

    Please advise..

    Reply
    1. avnika

      hi shalini…i am also getting the same error. do u got ant solution for this error?

      Reply
      1. Tushar Sarde Post author

        Something is wrong in your configuration files, please verify configurations again!

        Reply
        1. avnika

          hi tushar,
          is this configuration correct?

          @rem The java implementation to use. Required.
          @rem set JAVA_HOME=%JAVA_HOME%
          set JAVA_HOME=C:\Program Files (x86)\Java\jdk1.7.0_25

          Reply
          1. Tushar Sarde Post author

            download my conf folder from my github account and refer same file.
            I think its because of space between “Program Files (x86)”

            Reply
          2. Tushar Sarde Post author

            See this ..

            Progra~1 = ‘Program Files’
            Progra~2 = ‘Program Files(x86)’

            set JAVA_HOME=C:\Progra~2\Java\jdk1.7.0_25

            set above path and try

            Reply
              1. Tushar Sarde Post author

                did you use this path

                set JAVA_HOME=C:\Progra~2\Java\jdk1.7.0_25

                and command is

                hdfs namenode -format

                don’t add extra – after format

                Reply
    2. Tushar Sarde Post author

      Something is wrong in your configuration files, please verify configurations again!

      Reply
  5. Rohan

    Hi Tushar,

    Thanks for the detail information. After all steps performed when I performed the step: start-all.cmd I get error
    The script is Deprecated instead use start-dfs.cmd and start-yarn.cmd. And when start-dfs.cmd is executed I get error as The system cannot find the file hadoop.
    Please help me on this. Thanks

    Reply
  6. Lucky Okehigbemen

    Thanks a lot, the information is indeed helpful. “In next tutorial we will see how to run mapreduce programs in windows using eclipse and this hadoop setup” where can i find it?

    Reply
  7. Khushbu Murarka

    Hi Tushar,
    i have gone through the complete article, everything works fine but when i try to give jps command through the Terminal it says “jps’ is not recognized as an internal or external command,
    operable program or batch file.” also localhost:8088 works but localhost:50070 does not.
    can you please help me with this?

    Reply
    1. Shriya

      Hi Khushbu,
      Please check your hdfs-site.xml and make sure you add the path to your name node and data node, as is on your system. Separate directories will automatically get created once you successfully run localhost:50070.

      Reply
  8. Manish

    Hi Tusar, Thanks for the post, very helpful. Although I am stuck at one place. I formatted the namenode correctly but after that when I execute “start-dfs.cmd”, it doesn’t run and says “Window cannot find ‘hadoop,.Make sure you typed the name correctly..”. I am not sure what’s wrong. Could you please help me with this?
    Thanks!!

    Reply
  9. Nagendra

    Hi,

    When i try to run from eclipse i am getting unsatisfied link error. Can you please help me on this. I have downloaded your bin folder and still i am getting this error.

    Reply
  10. Gourav Soni

    It was an awesome tutorial. thanks a lot.
    could you share link for next tutorial link to start MapReduce ?

    Reply
  11. Akanksha

    I did all the configurations as you suggested.On jps below is output (indicating namenode, datanode and nodemanager are not getting started) :
    5604 ResourceManager
    5200 Jps

    Below is error in logs:
    2017-05-01 18:41:30,482 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
    java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:609)
    Please help.

    Reply
    1. Gourav Soni

      Check you hdfs-site.xml & yarn-site.xml edited properly.

      Reply
    2. Tushar Sarde Post author

      As Gourav mentioned, something is wrong in your configuration file!

      Reply
      1. Vikas

        Hi Tushar,
        I have also landed up in the same error, checked all configurations, they are as mentioned in your tutorial, any help?

        Reply
  12. Varun

    I have installed Hadoop based on the guide provide. How can I use this as Hive and start service to create basic table and insert data to it.

    Reply
    1. Tushar Sarde Post author

      We need to install hive package explicitly

      Reply
  13. Varun

    Very much helpful and easy process. Can I have the next topic on how can I start using this also does this support zappline ?

    Reply
  14. Padma

    your information is very helpful to understand for how install hadoop on window.but hadoop-2.7.1 link download failed when i trying to install.is it has any other to download.

    Reply
  15. Balajee

    Very Good Information, it helps me in finishing the Hadoop installation on Win 10 Quickly.

    Reply
  16. shivaprakash

    it very helpful 🙂 please if you have spatial hadoop implimentation in windows . please share

    Reply
  17. Karthik

    It was really helpful. After a lot of searching, I was able to get it running from the link.

    Thanks a lot.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *