Apache Hadoop 2.x Installation

1.    Download the tar ball from apache download mirrors.

2.    Extract the tar ball. It will create a directory hadoop-2.7.2 under current directory.

[NEON-RnD root@os3 lib]# tar xvzf hadoop-2.7.2.tar.gz

3.    Create a symbolic link named hadoop in /usr/lib directory pointing to Hadoop home. Say /usr/lib/Hadoop-2.7.2.

[NEON-RnD root@os3 lib]# ln -sf /root/lib/hadoop-2.7.2 /usr/lib/Hadoop

4.    Edit the ~/.bashrc file by following configurations.

export HADOOP_HOME=/usr/lib/hadoop

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/Hadoop

5.    Create a Temp folder in HADOOP_HOME

[NEON-RnD root@os3 libexec]# mkdir -p $HADOOP_HOME/tmp

6.    Edit Hadoop configuration files.

6.1./usr/lib/hadoop/etc/hadoop/hadoop-env.sh

[NEON-RnD root@os3 ~]# vi /usr/lib/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_67

6.2./usr/lib/hadoop/libexec/hadoop-config.sh

[NEON-RnD root@os3 ~]# vi /usr/lib/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_67

6.3. /usr/lib/hadoop/etc/hadoop/yarn-env.sh

[NEON-RnD root@os3 libexec]# vi /usr/lib/hadoop/etc/hadoop/yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_67

export HADOOP_HOME=/usr/lib/hadoop

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/Hadoop

6.4./usr/lib/hadoop/etc/hadoop/core-site.xml

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/core-site.xml

<configuration>

  <property>

    <name>fs.default.name</name>

    <value>hdfs://os3.flytxt.com:9000</value>

  </property>

  <property>

    <name>hadoop.tmp.dir</name>

    <value>/usr/lib/hadoop/tmp</value>

  </property>

</configuration>

6.5./usr/lib/hadoop/etc/hadoop/hdfs-site.xml

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/hdfs-site.xml

<configuration>

   <property>

     <name>dfs.replication</name>

     <value>2</value>

   </property>

   <property>

     <name>dfs.permissions</name>

     <value>false</value>

   </property>

</configuration>

6.6./usr/lib/hadoop/etc/hadoop/mapred-site.xml

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/mapred-site.xml

<configuration>

 <property>

   <name>mapreduce.framework.name</name>

   <value>yarn</value>

 </property>

</configuration>

6.7./usr/lib/hadoop/etc/hadoop/yarn-site.xml

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->

  <property>

    <name>yarn.nodemanager.aux-services</name>

    <value>mapreduce_shuffle</value>

  </property>

  <property>

    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

    <value>org.apache.hadoop.mapred.ShuffleHandler</value>

  </property>

  <property>

    <name>yarn.resourcemanager.resource-tracker.address</name>

    <value>os3.flytxt.com:8025</value>

  </property>

  <property>

    <name>yarn.resourcemanager.scheduler.address</name>

    <value>os3.flytxt.com:8030</value>

  </property>

  <property>

    <name>yarn.resourcemanager.address</name>

    <value>os3.flytxt.com:8040</value>

  </property>

</configuration>

6.8./usr/lib/hadoop/etc/hadoop/slaves

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/slaves

os3.flytxt.com

os4.flytxt.com

7.    Format the namenode and datanode.

[NEON-RnD root@os3 tmp]# hdfs namenode –format

[NEON-RnD root@os4 hadoop]# hdfs datanode  -format

8.    Start the daemons using following commands

[NEON-RnD root@os3 hadoop]# hadoop-daemon.sh start namenode

[NEON-RnD root@os3 hadoop]# hadoop-daemons.sh start datanode

[NEON-RnD root@os3 hadoop]# yarn-daemon.sh start resourcemanager

[NEON-RnD root@os3 hadoop]# yarn-daemons.sh start nodemanager

[NEON-RnD root@os3 hadoop]# mr-jobhistory-daemon.sh start historyserver

9.    Check the process started in each node using the jps command.

[NEON-RnD root@os3 hadoop]# for server in {'os3','os4'};do echo -e "$server\n***";ssh $server "jps -m | egrep -i 'namenode|datanode|resourcemanager|nodemanager|historyserver'";done

os3

***

28948 DataNode

28784 NameNode

29612 JobHistoryServer

29414 NodeManager

29088 ResourceManager

os4

***

6215 DataNode

6336 NodeManager