Home > Store

Self-Service Linux: Mastering the Art of Problem Determination

Register your product to gain access to bonus material or receive a coupon.

Self-Service Linux: Mastering the Art of Problem Determination


  • Sorry, this book is no longer in print.
Not for Sale



The Linux professionals' guide to effectively and efficiently diagnosing software problems and system crashes in the Linux environment.

° Learn to master use the most important problem determination tools that Linux has to offer

° Practical coverage of strace, the /proc filesystem, debugging options for use in compilation, and debugging with gdb

° In-depth coverage of kernel debugging techniques, from adding traps to using kdb


  • Copyright 2006
  • Dimensions: 7" x 9-1/4"
  • Pages: 456
  • Edition: 1st
  • Book
  • ISBN-10: 0-13-147751-X
  • ISBN-13: 978-0-13-147751-3

The biggest factor in a company's decision to go with Linux is overcoming thesupport issues. Adopting Linux successfully means taking on a differentapproach to support. Since Linux is open source, there are different methodsto get the support that companies need. This includes deeper skills on how toinvestigate problems, how to get help from the open source community (i.e.how to ask for help on usenet) and how to resolve problems in-house withoutthe extra cost of a consultant. This book should be an essential part of everycompany's Linux adoption plan to keep the total cost of ownership (TCO)down and improve the ROI of their Linux strategy. It is also a book thatadvanced Linux professionals running their own Linux systems will be able touse to troubleshoot. This book gives the staff the basics they need to diagnose most problems that they will face and will go into the nitty-gritty on the toughest problems. It also points users to the appropriate resources so that they may get help quickly.



Untitled Document Download Self-Service Linux® in PDF

Source Code

Download the Source Code file for this book.

Sample Content

Online Sample Chapter

Best Practices and Initial Investigation Techniques for Diagnosing Linux Problems

Table of Contents


1. Best Practices and Initial Investigation.

2. strace and System Call Tracing Explained.

3. The /proc Filesystem.

4. Compiling.

5. The Stack.

6. The GNU Debugger (GDB).

7. Linux System Crashes and Hangs.

8.: Kernel Debugging with KDB.

9.: ELF: Executable and Linking Format.

Appendix A: The Toolbox.

Appendix B: Data Collection Script.



Untitled Document Linux is the ultimate choice for home and business users. It is powerful, as stable as any commercial operating system, secure and best of all, it is open source. One of the biggest deciding factors for whether to use Linux at home or for your business can be service and support. Since Linux is developed by thousands of volunteers from around the world, it is not always clear who to turn to when something goes wrong

In the true spirit of Linux, there is a slightly different approach to support than the commercial norm. After all, Linux represents an unparalleled community of experts, it includes industry leading problem determination tools and of course, the product itself includes the source code. These resources are in addition to the professional Linux support services that are available from companies such as IBM and the various Linux vendors such as Redhat and SuSE. Making the most of these additional resources is called "self-service" and is the main topic covered by this book.

Self-service on Linux means different things to different people. For those who use Linux at home, it means a more enjoyable Linux experience. For those who use Linux at work, being able to quickly and effectively diagnose problems on Linux can increase their value as an employee as well as their marketability. For corporate leaders deciding whether to adopt Linux as part of the corporate strategy, self-service for Linux means reduced operation costs and increased Return On Investment (ROI) for any Linux adoption strategy. Regardless of what type of Linux user you are, it is important to make the most of your Linux experience and investment.

What is This Book About?

In a nutshell, this book is about effectively and efficiently diagnosing problems that occur in the Linux environment. It covers good investigation practices, how to use the information and resources on the Internet and then dives right into detail describing how to use the most important problem determination tools that Linux has to offer.

Chapter 1 is like a crash course on effective problem determination practices which will help you to diagnose problems like an expert. It covers where and how to look for information on the Internet as well as how to start investigating common types of problems.

Chapter 2 covers strace which is arguably the most frequently used problem determination tool on Linux. This chapter includes both practical usage information as well as details about how strace works. It also includes source code for a simple strace tool and details about how the underlying functionality works with the kernel through the ptrace interface.

Chapter 3 is about the /proc file system which contains a wealth of information about the hardware, kernel and processes that are running on the system. This purpose of this chapter is to point out and examine some of the more advanced features and tricks primarily related to problem determination and system diagnosis. For example, the chapter covers how to use the SysRq Kernel Magic hotkey with /proc/sys/kernel/sysrq.

Chapter 4 provides detailed information about compiling. Why does a book about debugging on Linux include a chapter about compiling? Well, the beginning of this preface mentioned that diagnosing problems on Linux is different than that of commercial environments. The main reason behind this is that the source code is freely available for all of the open sources tools and the operating system itself. This chapter provides vital information whether you need to recompile an open source application with debug information (as is often the case), whether you need to generate an assembly language listing for a tough problem (e.g., to find the line of code for a trap) or whether you run into a problem while recompiling the Linux kernel itself.

Chapter 5 covers intimate details about the stack, one of the most important and fundamental concepts of a computer system. Besides explaining all the gory details about the structure of a stack (which is pretty much required knowledge for any Linux expert), the chapter also includes and explains source code that can be used by the readers to generate stack traces from within their own tools and applications. The code examples are not only useful to illustrate how the stack works but they can save real time and debugging effort when included as part of an application's debugging facilities.

Chapter 6 takes an in-depth and detailed look at debugging applications with GDB and includes an overview of the DDD graphical user interface. Linux has an advantage over most other operating systems in that it includes a feature rich debugger, GDB, for free. Debuggers can be used to debug many types of problems and since GDB is free, it is well worth the effort to understand the basic and more advanced features. This chapter covers hard to find details about debugging C++ applications, threaded applications as well as numerous best practices. Have you ever spawned an xterm to attach to a process with gdb? This chapter will show you how any why!

Chapter 7 covers a detailed overview of system crashes and hangs. With proprietary operating systems a system crash or hang almost certainly requires you to call the OS vendor for help. However, with Linux the end user can debug a kernel problem on their own or at least identify key information to search for known problems. If you do need to get an expert involved, knowing what to collect will help you to get the right data quickly for a fast diagnosis. This chapter describes everything from how to attach a serial console to how to find the line of code for a kernel trap (an "Oops"). For example, the chapter provides step by step details for how to manually add a trap in the kernel and then debug it to find the resulting line of code.

Chapter 8 covers more details about debugging the kernel or debugging with the kernel debugger kdb. The chapter covers how to configure and enable kdb on your system as well as some practical commands that most Linux users can use without being a kernel expert. For example, this chapter will show you how to find out what a process is doing from within the kernel which can be particularly useful if the process is hung and not killable.

Chapter 9 is a detailed, head-on look at ELF (Executable and Linking Format). The details behind ELF are often ignored or just assumed to work. This is really unfortunate since a thorough understanding of ELF can lead to a whole new world of debugging techniques. The chapter covers intimate but practical details of the underlying ELF file format as well as tips and tricks that few people know. There is even sample code and step by step instructions for how to override functions using LD_PRELOAD and how to use the global offset table and the gdb debugger to intercept functions manually and redirect them to debug versions.

Appendix A is a tool box which outlines the most useful tools, facilities and files on Linux. For each tool, there is a description of when it is useful and where to get the latest copy.

Appendix B includes a production ready data collection script that is especially useful for mission critical systems or those who remotely support customers on Linux. The data collection script alone can save many hours or even days for debugging a remote problem.

Lastly, as we wrote this book it became clear to us that we were covering the right information. Reviewers often commented about how they were able to use the information immediately to solve real problems. Not the problems that may come in the future or may have happened in the past but real problems that people were actually struggling with when they reviewed the chapters. We also found ourselves referring to the content of the book to help solve problems as they came up. We hope you find it as useful as it has been to those who have read it thus far.

Who is This Book For?

This book has useful information for any Linux user but is certainly geared more towards the Linux professional. This includes Linux power users, Linux administrators, developers that write software for Linux and support staff that support products on Linux.

Readers that casually use Linux at home will also benefit as long as they either have a basic understanding of Linux or are at least willing to learn more about it, the latter being most important.

Lastly, as Linux increases in popularity, there are many seasoned experts who are facing the challenge of translating their knowledge and experience to the Linux platform. Many are already experts with one or more operating systems except that they lack specific knowledge about the various command line incantations or ways to interpret their knowledge for Linux. This book will help such experts to quickly adapt their existing skill set and apply it affectively on Linux.

This power packed book contains real industry experience on many topics and very hard to find information. Without a doubt, it is a must have for any developer, tester, support analyst or anyone that uses Linux.


Anyone who has written a book will agree that it takes an enormous amount of effort. Yes, there is a lot of work for the authors but without the many key people behind the scenes, writing a book would be nearly impossible. We would like to thank all of the people who reviewed, supported, contributed or otherwise made this book possible.

First we would like to thank the reviewers for their time, patience and valuable feedback. Besides the typos, grammatical errors and technical omissions, in many cases the reviewers allowed us to see other vantage points which in turn helped to make the content more well-rounded and complete. In particular, we would like to thank Richard Moore for reviewing the technical content of many chapters, Robert Haskins for being so thorough with his reviews and comments, Mel Gorman for his valuable feedback on the ELF (Executable and Linking Format) chapter, Scott Dier for his many valuable comments, Jan Kritter for reviewing pretty much the entire book, Joyce Coleman, Ananth Narayan, Pascale Stephenson, Ben Elliston, Hien Nguyen, Jim Keniston as well as the IBM Linux Technology Center for their valuable feedback. We would also like to thank the excellent engineers from SuSE for helping to answer many deep technical questions, especially Andi Kleen, Frank Balzer, and Michael Matz.

We would especially like to thank our wives and family for the support, encouragement and giving us the time to work on this book. Without their support, this book would have never gotten past the casual conversation we had about possibly writing a book many months ago. We truly appreciate the sacrifices that they have made to allow us to finish this book.

Lastly, we would like to thank the open source community as a whole. The open source movement is a truly remarkable phenomenon that has and will continue to raise the bar for computing at home or for commercial environments. Our thanks to the open source community is not specifically for this book but rather for their tireless dedication and technical prowess that make Linux and all open source products a reality. It is our hope that the content in this book will encourage others to adopt, use or support open source products and of course Linux. Every little bit helps.

Thanks for reading this book.


The history and evolution of the Linux operating system is fascinating and certainly still being written with new twists popping up all the time. Linux itself comprises only the kernel of the whole operating system. Granted, this is the single most important part, but everything else surrounding the Linux kernel is made up mostly of GNU free software. There are two major things that GNU software and the Linux kernel have in common. The first is that the source code for both is freely accessible. The second is that they have been developed and continue to be developed by many thousands of volunteers throughout the world, all connecting and sharing ideas and work through the Internet. Many refer to this collaboration of people and resources as the Open Source Community.

The Open Source Community is much like a distributed development team with skills and experience spanning many different areas of computer science. The source code that is written by the Open Source Community is available for anyone and everyone to see. Not only can this make problem determination easier, having such a large and diverse group of people looking at the code can reduce the number of defects and improve the security of the source code. Open Source Software is open to innovations as much as criticism, both helping to improve the quality and functionality of the software.

One of the most common concerns about adopting Linux is service and support. However, Linux has the Open Source Community, a wide range of freely available problem determination tools, the source code and the Internet itself as a source of information including numerous sites and newsgroups dedicated to Linux. It is important for every Linux user to understand the resources and tools that are available to help them diagnose problems. That is the purpose of this book. It is not intended to be a replacement to a support contract nor does it require one. If you have one, this book is an enhancement that will be sure to help you make the most of your support contract.


Download the Index file related to this title.


Submit Errata

More Information

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


This site is not directed to children under the age of 13.


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020