Home > Store > Data > Data Analytics and Visualization

Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with Apache Hadoop 2

Register your product to gain access to bonus material or receive a coupon.

Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with Apache Hadoop 2

Best Value Purchase

Book + eBook Bundle

  • Your Price: $43.19
  • List Price: $71.98
  • Includes EPUB, MOBI, and PDF
  • About eBook Formats
  • This eBook includes the following formats, accessible from your Account page after purchase:

    ePub EPUB The open industry format known for its reflowable content and usability on supported mobile devices.

    MOBI MOBI The eBook format compatible with the Amazon Kindle and Amazon Kindle applications.

    Adobe Reader PDF The popular standard, used most often with the free Adobe® Reader® software.

    This eBook requires no passwords or activation to read. We customize your eBook by discretely watermarking it with your name, making it uniquely yours.

More Purchase Options

Book

  • Your Price: $31.99
  • List Price: $39.99
  • Usually ships in 24 hours.

eBook (Watermarked)

  • Your Price: $25.59
  • List Price: $31.99
  • Includes EPUB, MOBI, and PDF
  • About eBook Formats
  • This eBook includes the following formats, accessible from your Account page after purchase:

    ePub EPUB The open industry format known for its reflowable content and usability on supported mobile devices.

    MOBI MOBI The eBook format compatible with the Amazon Kindle and Amazon Kindle applications.

    Adobe Reader PDF The popular standard, used most often with the free Adobe® Reader® software.

    This eBook requires no passwords or activation to read. We customize your eBook by discretely watermarking it with your name, making it uniquely yours.

Description

  • Copyright 2014
  • Dimensions: 7" x 9-1/8"
  • Pages: 400
  • Edition: 1st
  • Book
  • ISBN-10: 0-321-93450-4
  • ISBN-13: 978-0-321-93450-5

“This book is a critically needed resource for the newly released Apache Hadoop 2.0, highlighting YARN as the significant breakthrough that broadens Hadoop beyond the MapReduce paradigm.”
—From the Foreword by Raymie Stata, CEO of Altiscale


The Insider’s Guide to Building Distributed, Big Data Applications with Apache Hadoop™ YARN

Apache Hadoop is helping drive the Big Data revolution. Now, its data processing has been completely overhauled: Apache Hadoop YARN provides resource management at data center scale and easier ways to create distributed applications that process petabytes of data. And now in Apache Hadoop™ YARN, two Hadoop technical leaders show you how to develop new applications and adapt existing code to fully leverage these revolutionary advances.

YARN project founder Arun Murthy and project lead Vinod Kumar Vavilapalli demonstrate how YARN increases scalability and cluster utilization, enables new programming models and services, and opens new options beyond Java and batch processing. They walk you through the entire YARN project lifecycle, from installation through deployment.

You’ll find many examples drawn from the authors’ cutting-edge experience—first as Hadoop’s earliest developers and implementers at Yahoo! and now as Hortonworks developers moving the platform forward and helping customers succeed with it.

Coverage includes

  • YARN’s goals, design, architecture, and components—how it expands the Apache Hadoop ecosystem
  • Exploring YARN on a single node 
  • Administering YARN clusters and Capacity Scheduler 
  • Running existing MapReduce applications 
  • Developing a large-scale clustered YARN application 
  • Discovering new open source frameworks that run under YARN

Extras

Author's Site

Please visit the author's site here

Sample Content

Online Sample Chapters

Apache Hadoop YARN Install Quick Start

Apache Hadoop YARN: A Brief History and Rationale

Excerpt

Preface to Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with Apache Hadoop 2

Sample Pages

Download the sample pages (includes Chapters 2 and 3 and Index)

Table of Contents

Foreword by Raymie Stata xiii
Foreword by Paul Dix xv
Preface xvii
Acknowledgments xxi
About the Authors xxv

Chapter 1: Apache Hadoop YARN: A Brief History and Rationale 1
Introduction 1
Apache Hadoop 2
Phase 0: The Era of Ad Hoc Clusters 3
Phase 1: Hadoop on Demand 3
Phase 2: Dawn of the Shared Compute Clusters 9
Phase 3: Emergence of YARN 18
Conclusion 20

Chapter 2: Apache Hadoop YARN Install Quick Start 21
Getting Started 22
Steps to Configure a Single-Node YARN Cluster 22
Run Sample MapReduce Examples 30
Wrap-up 31

Chapter 3: Apache Hadoop YARN Core Concepts 33
Beyond MapReduce 33
Apache Hadoop MapReduce 35
Apache Hadoop YARN 38
YARN Components 39
Wrap-up 42

Chapter 4: Functional Overview of YARN Components 43
Architecture Overview 43
ResourceManager 45
YARN Scheduling Components 46
Containers 49
NodeManager 49
ApplicationMaster 50
YARN Resource Model 50
Managing Application Dependencies 53
Wrap-up 57

Chapter 5: Installing Apache Hadoop YARN 59
The Basics 59
System Preparation 60
Script-based Installation of Hadoop 2 62
Script-based Uninstall 68
Configuration File Processing 68
Configuration File Settings 68
Start-up Scripts 71
Installing Hadoop with Apache Ambari 71
Wrap-up 84

Chapter 6: Apache Hadoop YARN Administration 85
Script-based Configuration 85
Monitoring Cluster Health: Nagios 90
Real-time Monitoring: Ganglia 97
Administration with Ambari 99
JVM Analysis 103
Basic YARN Administration 106
Wrap-up 114

Chapter 7: Apache Hadoop YARN Architecture Guide 115
Overview 115
ResourceManager 117
NodeManager 127
ApplicationMaster 138
YARN Containers 148
Summary for Application-writers 150
Wrap-up 151

Chapter 8: Capacity Scheduler in YARN 153
Introduction to the Capacity Scheduler 153
Capacity Scheduler Configuration 155
Queues 156
Hierarchical Queues 156
Queue Access Control 159
Capacity Management with Queues 160
User Limits 163
Reservations 166
State of the Queues 167
Limits on Applications 168
User Interface 169
Wrap-up 169

Chapter 9: MapReduce with Apache Hadoop YARN 171
Running Hadoop YARN MapReduce Examples 171
MapReduce Compatibility 181
The MapReduce ApplicationMaster 181
Calculating the Capacity of a Node 182
Changes to the Shuffle Service 184
Running Existing Hadoop Version 1 Applications 184
Running MapReduce Version 1 Existing Code 187
Advanced Features 188
Wrap-up 190

Chapter 10: Apache Hadoop YARN Application Example 191
The YARN Client 191
The ApplicationMaster 208
Wrap-up 226

Chapter 11: Using Apache Hadoop YARN Distributed-Shell 227
Using the YARN Distributed-Shell 227
Internals of the Distributed-Shell 232
Wrap-up 240

Chapter 12: Apache Hadoop YARN Frameworks 241
Distributed-Shell 241
Hadoop MapReduce 241
Apache Tez 242
Apache Giraph 242
Hoya: HBase on YARN 243
Dryad on YARN 243
Apache Spark 244
Apache Storm 244
REEF: Retainable Evaluator Execution Framework 245
Hamster: Hadoop and MPI on the Same Cluster 245
Wrap-up 245

Appendix A: Supplemental Content and Code Downloads 247
Available Downloads 247

Appendix B: YARN Installation Scripts 249
install-hadoop2.sh 249
uninstall-hadoop2.sh 256
hadoop-xml-conf.sh 258

Appendix C: YARN Administration Scripts 263
configure-hadoop2.sh 263

Appendix D: Nagios Modules 269
check_resource_manager.sh 269
check_data_node.sh 271
check_resource_manager_old_space_pct.sh 272

Appendix E: Resources and Additional Information 277

 

Appendix F: HDFS Quick Reference 279
Quick Command Reference 279

Index 287

Updates

Submit Errata

More Information

Buy Together and Save

Total for all: $211.98 Add to cart

ONE MONTH ACCESS!

WITH PURCHASE


Get unlimited 30-day access to thousands of Books & Training Videos about technology, professional development and digital media If you continue your subscription after your 30-day trial, you can receive 30% off a monthly subscription to the Safari Library for up to 12 months.