By | August 10, 2015

Hadoop installation on windows without cygwin in 10 mints – Hadoop installation on windows 7 or 8

Download
Before starting make sure you have this two softwares

Extract downloaded tar file

Configuration
Step 1 – Windows path configuration
set HADOOP_HOME path in enviornment variable for windows
Right click on my computer > properties > advanced system settings > advance tab > environment variables > click on new

Set hadoop bin directory path
Find path variable in system variable > click on edit > at the end insert ‘; (semicolon)’ and paste path upto hadoop bin directory in my case it’s a

F:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/bin


Step 2 – Hadoop configuration
Edit hadoop-2.7.1/etc/hadoop/core-site.xml, paste the following lines and save it.

<configuration>
<property>
       <name>fs.defaultFS</name>
       <value>hdfs://localhost:9000</value>
   </property>
</configuration>

Edit hadoop-2.7.1/etc/hadoop/mapred-site.xml, paste the following lines and save it.

<configuration>
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
</configuration>

Edit hadoop-2.7.1/etc/hadoop/hdfs-site.xml, paste the following lines and save it, please create data folder somewhere and in my case i have created it in my HADOOP_HOME directory

<configuration>
<property>
       <name>dfs.replication</name>
       <value>1</value>
   </property>
   <property>
       <name>dfs.namenode.name.dir</name>
       <value>/f:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/data/namenode</value>
   </property>
   <property>
       <name>dfs.datanode.data.dir</name>
     <value>/f:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/data/datanode</value>
   </property>
</configuration>

Edit hadoop-2.7.1/etc/hadoop/yarn-site.xml, paste the following lines and save it.

<configuration>
   <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
   </property>
   <property>
       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
       <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>
</configuration>

Edit hadoop-2.7.1/etc/hadoop/hadoop-env.cmd, comment existing %JAVA_HOME% using @rem at start, give proper path and save it. (my jdk is in program files to avoid spaces i gave PROGRA~1)



Demo

Step 3 – Start everything

Very Important step

Before starting everything you need to add some [dot].dll and [dot].exe files of windows please download bin folder from my github repository – sardetushar_gitrepo_download

bin folder – this contains .dll and .exe file (winutils.exe for hadoop 2.7.1)

Now delete you existing bin folder and replace with new one (downloaded from my repo )

( my github ) etc folder is just given for reference you need to modify your configuration parameters according to your environment path

Open cmd and type ‘hdfs namenode -format’ – after execution you will see below logs

Open cmd and point to sbin directory and type ‘start-all.cmd

F:Hortanwork1gbhadoopramSoftwarehadoop-2.7hadoop-2.7.1sbin>start-all.cmd

It will start following process

Namenode

Datanode

YARN resourcemanager

YARN nodemanager


JPS – to see services are running
open cmd and type – jps (for jps make sure your java path is set properly)

GUI
Step 4 – namenode GUI, resourcemanager GUI
Resourcemanager GUI address – http://localhost:8088

Namenode GUI address – http://localhost:50070

In next tutorial we will see how to run mapreduce programs in windows using eclipse and this hadoop setup




Share this knowledge ! Join us on Facebook ! Now Whatsapp sharing is supportable ! TooDey Inc.

15 thoughts on “

  1. lee

    hi, I followed the process, but my resource namger shutting down automatically.
    Can you tell me why?

    17/12/11 12:38:11 WARN amlauncher.ApplicationMasterLauncher: org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning.
    17/12/11 12:38:11 INFO ipc.Server: Stopping server on 8030
    17/12/11 12:38:11 INFO ipc.Server: Stopping IPC Server listener on 8030
    17/12/11 12:38:11 INFO ipc.Server: Stopping IPC Server Responder
    17/12/11 12:38:11 INFO ipc.Server: Stopping server on 8031
    17/12/11 12:38:11 INFO ipc.Server: Stopping IPC Server listener on 8031
    17/12/11 12:38:11 INFO ipc.Server: Stopping IPC Server Responder
    17/12/11 12:38:11 ERROR event.EventDispatcher: Returning, interrupted : java.lang.InterruptedException
    17/12/11 12:38:11 INFO util.AbstractLivelinessMonitor: NMLivelinessMonitor thread interrupted
    17/12/11 12:38:11 INFO event.AsyncDispatcher: AsyncDispatcher is draining to stop, ignoring any new events.
    17/12/11 12:38:11 INFO util.AbstractLivelinessMonitor: org.apache.hadoop.yarn.server.resourcemanager.rmapp.monitor.RMAppLifetimeMonitor thread interrupted
    17/12/11 12:38:11 INFO util.AbstractLivelinessMonitor: AMLivelinessMonitor thread interrupted
    17/12/11 12:38:11 ERROR delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
    17/12/11 12:38:11 INFO util.AbstractLivelinessMonitor: org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer thread interrupted
    17/12/11 12:38:11 INFO util.AbstractLivelinessMonitor: AMLivelinessMonitor thread interrupted
    17/12/11 12:38:11 INFO impl.MetricsSystemImpl: Stopping ResourceManager metrics system…
    17/12/11 12:38:11 INFO impl.MetricsSystemImpl: ResourceManager metrics system stopped.
    17/12/11 12:38:11 INFO impl.MetricsSystemImpl: ResourceManager metrics system shutdown complete.
    17/12/11 12:38:11 INFO event.AsyncDispatcher: AsyncDispatcher is draining to stop, ignoring any new events.
    17/12/11 12:38:11 INFO resourcemanager.ResourceManager: Transitioned to standby state
    17/12/11 12:38:11 INFO resourcemanager.ResourceManager: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down ResourceManager at DESKTOP-1TI8I6M/10.0.0.105.

    Reply
    1. Tushar

      Something wrong in your configuration file or your system is not allowing to open resources manager port!
      Please verify the security settings.

      Reply
  2. Yuva Phani Kosuru

    Hi,

    Thanks you so much. It helped me a lot. I am getting error after launching start-all command. In node manager i am getting as below:

    `17/11/18 11:38:20 WARN localizer.ResourceLocalizationService: Permissions incorrectly set for dir /tmp/hadoop-Naveen/nm-local-dir/usercache, should be rwxr-xr-x, actual value = rwxrwxr-x
    17/11/18 11:38:20 INFO localizer.ResourceLocalizationService: Attempting to initialize /tmp/hadoop-Naveen/nm-local-dir
    17/11/18 11:38:22 WARN util.SysInfoWindows: Expected split length of sysInfo to be 11. Got 7
    17/11/18 11:38:22 WARN localizer.ResourceLocalizationService: Permissions incorrectly set for dir /tmp/hadoop-Naveen/nm-local-dir/usercache, should be rwxr-xr-x, actual value = rwxrwxr-x
    17/11/18 11:38:22 WARN localizer.ResourceLocalizationService: Failed to setup local dir /tmp/hadoop-Naveen/nm-local-dir, which was marked as good.
    org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions incorrectly set for dir /tmp/hadoop-Naveen/nm-local-dir/usercache, should be rwxr-xr-x, actual value = rwxrwxr-x
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1562)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationService.java:1530)`

    please help on this.

    Reply
  3. Arpan parikh

    I am getting below error while doing start-all.sh

    17/11/12 19:42:46 WARN util.SysInfoWindows: Expected split length of sysInfo to be 11. Got 7

    Reply
  4. Sumanth

    Thanks A Lot. Could you also provide some information about SQOOP, HIVE, SPARK and HBASE installation on Windows.

    Reply
  5. hsiwei yu

    Thanks Tushar,

    This is great! I just got hadoop 2.8.1 installed & seems to be working, on my Windows 10 laptop.

    Questions:
    1) What is jps? i don’t see in 2.8.1 sbin or bin directories.
    2) what’s the significance of your bin?
    sardetushar_gitrepo_download?
    3) what to do next, for learning hadoop?

    Thanks!

    Two minor changes for me,
    1) i copied your bin about 13 files (NOT replacing what’s already there) to original hadoop 2.8.1 bin. Didn’t take all of your bin directory.

    2) hdfs-site.xml, i changed to
    file:///f:/Hortanwork/1gbhadoopram .. ..
    from
    /f:/Hortanwork/1gbhadoopram .. ..
    then it worked for me.

    dfs.namenode.name.dir
    file:///f:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/data/namenode

    Reply
    1. Fayzan

      Can i install it on windows 10? Have you faced any issues installing it on windx10.

      Reply
  6. Juan

    Can someone help me with this error,
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.
    Error: Could not find or load main class J

    Where I can find the J class?

    Thanks for your respond 🙂

    Reply
  7. Alexey

    Hi! It was really useful for me, thanks a lot! Can we also install hbase same way?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *