Apache Hadoop 2.x Installation

1.    Download the tar ball from apache download mirrors.

2.    Extract the tar ball. It will create a directory hadoop-2.7.2 under current directory.

[NEON-RnD root@os3 lib]# tar xvzf hadoop-2.7.2.tar.gz

3.    Create a symbolic link named hadoop in /usr/lib directory pointing to Hadoop home. Say /usr/lib/Hadoop-2.7.2.

[NEON-RnD root@os3 lib]# ln -sf /root/lib/hadoop-2.7.2 /usr/lib/Hadoop

4.    Edit the ~/.bashrc file by following configurations.

export HADOOP_HOME=/usr/lib/hadoop

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/Hadoop

5.    Create a Temp folder in HADOOP_HOME

[NEON-RnD root@os3 libexec]# mkdir -p $HADOOP_HOME/tmp

6.    Edit Hadoop configuration files.

6.1./usr/lib/hadoop/etc/hadoop/hadoop-env.sh

[NEON-RnD root@os3 ~]# vi /usr/lib/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_67

6.2./usr/lib/hadoop/libexec/hadoop-config.sh

[NEON-RnD root@os3 ~]# vi /usr/lib/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_67

6.3. /usr/lib/hadoop/etc/hadoop/yarn-env.sh

[NEON-RnD root@os3 libexec]# vi /usr/lib/hadoop/etc/hadoop/yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_67

export HADOOP_HOME=/usr/lib/hadoop

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/Hadoop

6.4./usr/lib/hadoop/etc/hadoop/core-site.xml

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/core-site.xml

<configuration>

  <property>

    <name>fs.default.name</name>

    <value>hdfs://os3.flytxt.com:9000</value>

  </property>

  <property>

    <name>hadoop.tmp.dir</name>

    <value>/usr/lib/hadoop/tmp</value>

  </property>

</configuration>

6.5./usr/lib/hadoop/etc/hadoop/hdfs-site.xml

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/hdfs-site.xml

<configuration>

   <property>

     <name>dfs.replication</name>

     <value>2</value>

   </property>

   <property>

     <name>dfs.permissions</name>

     <value>false</value>

   </property>

</configuration>

6.6./usr/lib/hadoop/etc/hadoop/mapred-site.xml

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/mapred-site.xml

<configuration>

 <property>

   <name>mapreduce.framework.name</name>

   <value>yarn</value>

 </property>

</configuration>

6.7./usr/lib/hadoop/etc/hadoop/yarn-site.xml

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->

  <property>

    <name>yarn.nodemanager.aux-services</name>

    <value>mapreduce_shuffle</value>

  </property>

  <property>

    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

    <value>org.apache.hadoop.mapred.ShuffleHandler</value>

  </property>

  <property>

    <name>yarn.resourcemanager.resource-tracker.address</name>

    <value>os3.flytxt.com:8025</value>

  </property>

  <property>

    <name>yarn.resourcemanager.scheduler.address</name>

    <value>os3.flytxt.com:8030</value>

  </property>

  <property>

    <name>yarn.resourcemanager.address</name>

    <value>os3.flytxt.com:8040</value>

  </property>

</configuration>

6.8./usr/lib/hadoop/etc/hadoop/slaves

[NEON-RnD root@os3 tmp]# vi /usr/lib/hadoop/etc/hadoop/slaves

os3.flytxt.com

os4.flytxt.com

7.    Format the namenode and datanode.

[NEON-RnD root@os3 tmp]# hdfs namenode –format

[NEON-RnD root@os4 hadoop]# hdfs datanode  -format

8.    Start the daemons using following commands

[NEON-RnD root@os3 hadoop]# hadoop-daemon.sh start namenode

[NEON-RnD root@os3 hadoop]# hadoop-daemons.sh start datanode

[NEON-RnD root@os3 hadoop]# yarn-daemon.sh start resourcemanager

[NEON-RnD root@os3 hadoop]# yarn-daemons.sh start nodemanager

[NEON-RnD root@os3 hadoop]# mr-jobhistory-daemon.sh start historyserver

9.    Check the process started in each node using the jps command.

[NEON-RnD root@os3 hadoop]# for server in {'os3','os4'};do echo -e "$server\n***";ssh $server "jps -m | egrep -i 'namenode|datanode|resourcemanager|nodemanager|historyserver'";done

os3

***

28948 DataNode

28784 NameNode

29612 JobHistoryServer

29414 NodeManager

29088 ResourceManager

os4

***

6215 DataNode

6336 NodeManager

Install Apache Ambari In red hat Linux

1.    Install the epel repository:

[NEON-RnD root@os3 ~]# yum install epel-release

2.    Check ambari repository added in repo list.

[NEON-RnD root@os3 ~]# yum repolist

Loaded plugins: product-id, refresh-packagekit, security, subscription-manager

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

repo id                                      repo name         status

HDP-2.2                                    HDP                  191

HDP-UTILS-1.1.0.20                   HDP-UTILS         42

rhel66                                       RHEL 6.6           3,913

repolist: 4,146

3.    If repo is not added then execute the following command. It will add an ambari repository in “/etc/yum.repos.d/”.

[NEON-RnD root@os3 ~]# cd /etc/yum.repos.d/

[NEON-RnD root@os3 yum.repos.d]# wget http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.1.0/ambari.repo

4.    Install Ambari server using yum command.

[NEON-RnD root@os3 yum.repos.d]# yum install ambari-server

5.    Setup Ambari server using following steps. We need to configure the database properties, user configuration, and JDK configuration.

[NEON-RnD root@os3 ~]# ambari-server setup

Using python  /usr/bin/python2.6

Setup ambari-server

Checking SELinux...

SELinux status is 'disabled'

Customize user account for ambari-server daemon [y/n] (n)? y

Enter user account for ambari-server daemon (root):

Adjusting ambari-server permissions and ownership...

Checking firewall status...

Checking JDK...

Do you want to change Oracle JDK [y/n] (n)? y

[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7

[3] Custom JDK

==============================================================================

Enter choice (1): 3

WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.

WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.

Path to JAVA_HOME: /usr/java/jdk1.7.0_67

Validating JDK on Ambari Server...done.

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)? y

Configuring database...

==============================================================================

Choose one of the following options:

[1] - PostgreSQL (Embedded)

[2] - Oracle

[3] - MySQL

[4] - PostgreSQL

[5] - Microsoft SQL Server (Tech Preview)

==============================================================================

Enter choice (1): 3

Hostname (192.168.150.170):

Port (3306):

Database name (ambari):

Username (root):

Enter Database Password (bigdata):

Re-enter password:

Configuring ambari database...

Copying JDBC drivers to server resources...

Configuring remote database connection properties...

WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

Proceed with configuring remote database connection properties [y/n] (y)? y

Extracting system views...

.......

Adjusting ambari-server permissions and ownership...

Ambari Server 'setup' completed successfully.

6.       Ambari will create an SQL file named /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql. We need to create a database named ambari in MySQL following by executing this file. Also please ensure the necessary privileges has granted to the user configured.

[NEON-RnD root@os3 jdk1.7.0_67]# mysql -h 192.168.150.170 -u root -p

Enter password:

mysql> create database ambari;

Query OK, 1 row affected (0.00 sec)

mysql> mysql> grant all privileges on ambari.* to 'root'@'%' ;

Query OK, 0 rows affected (0.00 sec)

[NEON-RnD root@os3 ~]# mysql -h 192.168.150.170 -u root -p ambari < /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

7.       All configurations are done at this stage. It’s time to start the Ambari Server. Use following command to start the server.

[NEON-RnD root@os3 ~]# ambari-server start

Using python  /usr/bin/python2.6

Starting ambari-server

Ambari Server running with administrator privileges.

Organizing resource files at /var/lib/ambari-server/resources...

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Waiting for server start....................

Ambari Server 'start' completed successfully.

8.       The web interface of Ambari can be accessed from <hostname>:8080. Default username and password for logging ambari is admin/admin.