- Installation Platforms and Prerequisites
- Installing Hadoop
- Deploying Hadoop in the Cloud
- Summary
- Q&A
- Workshop
This chapter is from the book
This chapter is from the book
This chapter is from the book
Installing Hadoop
You have numerous options for installing Hadoop and setting up Hadoop clusters. As Hadoop is a top-level Apache Software Foundation (ASF) open source project, one method is to install directly from the Apache builds on http://hadoop.apache.org/. To do this you first need one or more hosts, depending upon the mode you wish to use, with appropriate hardware specifications, an appropriate operating system, and a Java runtime environment available (all of the prerequisites and considerations discussed in the previous section).
Once you have this, it is simply a matter of downloading and unpacking the desired release. There may be some additional configuration to be done afterwards, but then you simply start the relevant services (master and slave node daemons) on their designated hosts and you are up and running.
Non-Commercial Hadoop
Let’s deploy a Hadoop cluster using the latest Apache release now.
Try It Yourself: Installing Hadoop Using the Apache Release
In this exercise we will install a pseudo-distributed mode Hadoop cluster using the latest Hadoop release downloaded from hadoop.apache.org.
As this is a test cluster the following specifications will be used in our example:
Red Hat Enterprise Linux 7.2 (The installation steps would be similar using other Linux distributions such as Ubuntu)
2 CPU cores
8GB RAM
30GB HDD
hostname: hadoopnode0
Disable SELinux (this is known to cause issues with Hadoop):
$ sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
Disable IPv6 (this is also known to cause issues with Hadoop):
$ sudo sed -i "\$anet.ipv6.conf.all.disable_ipv6 = 1" /etc/sysctl.conf $ sudo sed -i "\$anet.ipv6.conf.default.disable_ipv6 = 1" /etc/sysctl.conf $ sudo sysctl -p
Reboot
Run the sestatus command to ensure SELinux is not enabled:
$ sestatus
Install Java. We will install the OpenJDK, which will install both a JDK and JRE:
$ sudo yum install java-1.7.0-openjdk-devel
Test that Java has been successfully installed by running the following command:
$ java -version
If Java has been installed correctly you should see output similar to the following:
java version "1.7.0_101" OpenJDK Runtime Environment (rhel-2.6.6.1.el7_2-x86_64..) OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
Note that depending upon which operating system you are deploying on, you may have a version of Java and a JDK installed already. In these cases it may not be necessary to install the JDK, or you may need to set up alternatives so you do not have conflicting Java versions.
Locate the installation path for Java, and set the JAVA_HOME environment variable:
Download Hadoop from your nearest Apache download mirror. You can obtain the link by selecting the binary option for the version of your choice at http://hadoop.apache.org/releases.html. We will use Hadoop version 2.7.2 for our example.
$ wget http://REPLACE_WITH_YOUR_MIRROR/hadoop-2.7.2.tar.gz
Unpack the Hadoop release, move it into a system directory, and set an environment variable from the Hadoop home directory:
$ tar -xvf hadoop-2.7.2.tar.gz $ mv hadoop-2.7.2 hadoop $ sudo mv hadoop/ /usr/share/ $ export HADOOP_HOME=/usr/share/hadoop
Create a directory which we will use as an alternative to the Hadoop configuration directory:
$ sudo mkdir -p /etc/hadoop/conf
Create a mapred-site.xml file (I will discuss this later) in the Hadoop configuration directory:
$ sudo cp $HADOOP_HOME/etc/hadoop/mapred-site.xml.template $HADOOP_HOME/etc/hadoop/mapred-site.xml
Add JAVA_HOME environment variable to hadoop-env.sh (file used to source environment variables for Hadoop processes):
$ sed -i "\$aexport JAVA_HOME=/REPLACE_WITH_YOUR_JDK_PATH/" $HADOOP_HOME/etc/hadoop/hadoop-env.sh
Substitute the correct path to your Java home directory as defined in Step 6.
Create a symbolic link between the Hadoop configuration directory and the /etc/hadoop /conf directory created in Step 10:
Create a logs directory for Hadoop:
$ mkdir $HADOOP_HOME/logs
Create users and groups for HDFS and YARN:
$ sudo groupadd hadoop $ sudo useradd -g hadoop hdfs $ sudo useradd -g hadoop yarn
Change the group and permissions for the Hadoop release files:
$ sudo chgrp -R hadoop /usr/share/hadoop $ sudo chmod -R 777 /usr/share/hadoop
Run the built in Pi Estimator example included with the Hadoop release.
$ cd $HADOOP_HOME $ sudo -u hdfs bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 16 1000
As we have not started any daemons or initialized HDFS, this program runs in LocalJobRunner mode (recall that I discussed this in Hour 2, “Understanding the Hadoop Cluster Architecture”). If this runs correctly you should see output similar to the following:
... Job Finished in 2.571 seconds Estimated value of Pi is 3.14250000000000000000
Now let’s configure a pseudo-distributed mode Hadoop cluster from your installation.
Use the vi editor to update the core-site.xml file, which contains important information about the cluster, specifically the location of the namenode:
$ sudo vi /etc/hadoop/conf/core-site.xml # add the following config between the <configuration> # and </configuration> tags: <property> <name>fs.defaultFS</name> <value>hdfs://hadoopnode0:8020</value> </property>
Note that the value for the fs.defaultFS configuration parameter needs to be set to hdfs://HOSTNAME:8020, where the HOSTNAME is the name of the NameNode host, which happens to be the localhost in this case.
Adapt the instructions in Step 17 to similarly update the hdfs-site.xml file, which contains information specific to HDFS, including the replication factor, which is set to 1 in this case as it is a pseudo-distributed mode cluster:
sudo vi /etc/hadoop/conf/hdfs-site.xml # add the following config between the <configuration> # and </configuration> tags: <property> <name>dfs.replication</name> <value>1</value> </property>
Adapt the instructions in Step 17 to similarly update the yarn-site.xml file, which contains information specific to YARN. Importantly, this configuration file contains the address of the resourcemanager for the cluster—in this case it happens to be the localhost, as we are using pseudo-distributed mode:
$ sudo vi /etc/hadoop/conf/yarn-site.xml # add the following config between the <configuration> # and </configuration> tags: <property> <name>yarn.resourcemanager.hostname</name> <value>hadoopnode0</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>
Adapt the instructions in Step 17 to similarly update the mapred-site.xml file, which contains information specific to running MapReduce applications using YARN:
$ sudo vi /etc/hadoop/conf/mapred-site.xml # add the following config between the <configuration> # and </configuration> tags: <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>
Format HDFS on the NameNode:
$ sudo -u hdfs bin/hdfs namenode -format
Enter [Y] to re-format if prompted.
Start the NameNode and DataNode (HDFS) daemons:
$ sudo -u hdfs sbin/hadoop-daemon.sh start namenode $ sudo -u hdfs sbin/hadoop-daemon.sh start datanode
Start the ResourceManager and NodeManager (YARN) daemons:
$ sudo -u yarn sbin/yarn-daemon.sh start resourcemanager $ sudo -u yarn sbin/yarn-daemon.sh start nodemanager
Use the jps command included with the Java JDK to see the Java processes that are running:
$ sudo jps
You should see output similar to the following:
2374 DataNode 2835 Jps 2280 NameNode 2485 ResourceManager 2737 NodeManager
Create user directories and a tmp directory in HDFS and set the appropriate permissions and ownership:
$ sudo -u hdfs bin/hadoop fs -mkdir -p /user/<your_user> $ sudo -u hdfs bin/hadoop fs -chown <your_user>:<your_user> /user/<your_user> $ sudo -u hdfs bin/hadoop fs -mkdir /tmp $ sudo -u hdfs bin/hadoop fs -chmod 777 /tmp
Now run the same Pi Estimator example you ran in Step 16. This will now run in pseudo-distributed mode:
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 16 1000
The output you will see in the console will be similar to that in Step 16. Open a browser and go to localhost:8088. You will see the YARN ResourceManager Web UI (which I discuss in Hour 6, “Understanding Data Processing in Hadoop”) (Figure 3.1):
Figure 3.1 YARN ResourceManager Web UI.
$ export JAVA_HOME=/usr/lib/jvm/REPLACE_WITH_YOUR_PATH/
$ sudo ln -s $HADOOP_HOME/etc/hadoop/* /etc/hadoop/conf/
)
Congratulations! You have just set up your first pseudo-distributed mode Hadoop cluster. We will use this cluster in other exercises in the book, so keep it available if you can.
Using a Commercial Hadoop Distribution
As I had discussed in Hour 1, “Introducing Hadoop,” the commercial Hadoop landscape is well established. With the advent of the ODPi (the Open Data Platform initiative), a once-numerous array of vendors and derivative distributions has been consolidated to a much simpler landscape which includes three primary pure-play Hadoop vendors:
Cloudera
Hortonworks
MapR
Importantly, enterprise support agreements and subscriptions can be purchased from the various Hadoop vendors for their distributions. Each vendor also supplies a suite of management utilities to help you deploy and manage Hadoop clusters. Let’s look at each of the three major pure play Hadoop vendors and their respective distributions.
Cloudera
Cloudera was the first mover in the commercial Hadoop space, establishing their first commercial release in 2008. Cloudera provides a Hadoop distribution called CDH (Cloudera Distribution of Hadoop), which includes the Hadoop core and many ecosystem projects. CDH is entirely open source.
Cloudera also provides a management utility called Cloudera Manager (which is not open source). Cloudera Manager provides a management console and framework enabling the deployment, management, and monitoring of Hadoop clusters, and which makes many administrative tasks such as setting up high availability or security much easier. The mix of open source and proprietary software is often referred to as open core. A screenshot showing Cloudera Manager is pictured in Figure 3.2.
)
Figure 3.2 Cloudera Manager.
As mentioned, Cloudera Manager can be used to deploy Hadoop clusters, including master nodes, slave nodes, and ecosystem technologies. Cloudera Manager distributes installation packages for Hadoop components through a mechanism called parcels. As Hadoop installations are typically isolated from public networks, Cloudera Manager, which is technically not part of the cluster and will often have access to the Internet, will download parcels and distribute these to new target hosts nominated to perform roles in a Hadoop cluster or to existing hosts to upgrade components.
Deploying a fully distributed CDH cluster using Cloudera Manager would involve the following steps at a high level:
Install Cloudera Manager on a host that has access to other hosts targeted for roles in the cluster.
Specify target hosts for the cluster using Cloudera Manager.
Use Cloudera Manager to provision Hadoop services, including master and slave nodes.
Cloudera also provides a Quickstart virtual machine, which is a pre-configured pseudo-distributed Hadoop cluster with the entire CDH stack, including core and ecosystem components, as well as a Cloudera Manager installation. This virtual machine is available for VirtualBox, VMware, and KVM, and works with the free editions of each of the virtualization platforms. The Cloudera Quickstart VM is pictured in Figure 3.3.
)
Figure 3.3 Cloudera Quickstart VM.
The Quickstart VM is a great way to get started with the Cloudera Hadoop offering. To find out more, go to http://www.cloudera.com/downloads.html.
More information about Cloudera is available at http://www.cloudera.com/.
Hortonworks
Hortonworks provides pure open source Hadoop distribution and a founding member of the open data platform initiative (ODPi) discussed in Hour 1. Hortonworks delivers a distribution called HDP (Hortonworks Data Platform), which is a complete Hadoop stack including the Hadoop core and selected ecosystem components. Hortonworks uses the Apache Ambari project to provide its deployment configuration management and cluster monitoring facilities. A screenshot of Ambari is shown in Figure 3.4.
)
Figure 3.4 Ambari console.
The simplest method to deploy a Hortonworks Hadoop cluster would involve the following steps:
Install Ambari using the Hortonworks installer on a selected host.
Add hosts to the cluster using Ambari.
Deploy Hadoop services (such as HDFS and YARN) using Ambari.
Hortonworks provides a fully functional, pseudo-distributed HDP cluster with the complete Hortonworks application stack in a virtual machine called the Hortonworks Sandbox. The Hortonworks Sandbox is available for VirtualBox, VMware, and KVM. The Sandbox virtual machine includes several demo applications and learning tools to use to explore Hadoop and its various projects and components. The Hortonworks Sandbox welcome screen is shown in Figure 3.5.
)
Figure 3.5 Hortonworks Sandbox.
You can download the Hortonworks Sandbox from http://hortonworks.com/products/sandbox/. More information about Hortonworks and HDP can be obtained from http://hortonworks.com/.
MapR
MapR delivers a “Hadoop-derived” software platform that implements an API-compatible distributed filesystem called MapRFS (the MapR distributed Filesystem). MapRFS has been designed to maximize performance and provide read-write capabilities not offered by native HDFS. MapR delivers three versions of their offering called the “Converged Data Platform.” These include:
M3 or “Converged Community Edition” (free version)
M5 or “Converged Enterprise Edition” (supported version)
M7 (M5 version that includes MapR’s custom HBase-derived data store)
Like the other distributions, MapR has a demo offering called the “MapR Sandbox,” which is available for VirtualBox or VMware. It is pictured in Figure 3.6.
)
Figure 3.6 MapR Sandbox VM.
The MapR Sandbox can be downloaded from https://www.mapr.com/products/mapr-sandbox-hadoop/download.
MapR’s management offering is called the MapR Control System (MCS), which offers a central console to configure, monitor and manage MapR clusters. It is shown in Figure 3.7.
)
Figure 3.7 MapR Control System (MCS).
Much more information about MapR and the Converged Data Platform is available at https://www.mapr.com/.