Home > Store

Critical Testing Processes: Plan, Prepare, Perform, Perfect

Register your product to gain access to bonus material or receive a coupon.

Critical Testing Processes: Plan, Prepare, Perform, Perfect


  • Sorry, this book is no longer in print.
Not for Sale


  • Copyright 2004
  • Dimensions: 7" x 9-1/4"
  • Pages: 608
  • Edition: 1st
  • Book
  • ISBN-10: 0-201-74868-1
  • ISBN-13: 978-0-201-74868-0

The advent of agile methodologies and test-driven development has brought software testing to the forefront of application development. Yet in today’s harried rush-to-market development environment, organizations must find a delicate balance between product release and product quality.

In Critical Testing Processes, the author distills knowledge gained from 20 years of testing experience into twelve critical processes. These include highly visible processes by which peers and management judge competence, and mission-critical processes in which performance affects the company’s profits and reputation.

After each process is introduced, the author demonstrates its use through an engaging case study. Instead of cumbersome regulations, this book provides checklists—lightweight, flexible tools for implementing process-oriented testing, gathering metrics, and making incremental process changes. By demonstrating critical processes in various organizational, operational, and technological contexts, this book shows readers how to:

  • Handle recurrent tests efficiently and consistently
  • Develop a cohesive, cooperative team that sidesteps redundancies
  • Build a reputation for reliability through the effective communication of test results
  • Define the focus of tests for maximum customer satisfaction and organizational success

    Because testing is a collaborative process with the participation of staff throughout the organization, the author discusses interpersonal and cultural issues in depth. This book also devotes ample coverage to the often-overlooked areas of planning and perfecting tests. Whatever your role in testing—from test engineering to managing hundreds of test engineers—Critical Testing Processes will offer valuable insights into what you do, why it’s important, and how you can perform better.


  • Sample Content

    Downloadable Sample Chapter

    Download the Sample Chapter related to this title.

    Table of Contents





    Book Scope.

    Who Should Read This Book?

    Demystifying Processes.

    Types and Relationships of Critical Testing Processes.

    Critical Testing Processes in Context.

    Sumatra, a Hypothetical Project.


    On Using This Book.

    Recognize Good Processes.

    Handle Challenges.

    Implement Improvements.

    To Your Testing Success!

    I. PLAN.

    1. Start with the Big Picture: Put the Test Role in the Broader Context.

    A Testing Process.

    Understanding Operational and Organizational Context.

    Introducing the Sumatra Project and the SpeedyWriter Test Team.

    Testing within the System Development Lifecycle.

    Organizing the Testers and the Testing.

    Management Dimensions.

    Beyond Testing Process Context.

    2. Focus the Effort: Analyze Quality Risks.

    A Quality Risk Analysis Process.

    Jamal Introduces the Team to Quality Risk Analysis.

    Recognize a Good Quality Risk Analysis Process.

    Involve the Right Participants.

    Employ the Right Technique.

    Separate the Vital Few from the Trivial Many.

    Find Potential Bugs before Testing.

    Start at the Right Time.

    Adapt to Change.

    Handle Challenges.

    Getting People to Get Real about Risk.

    Quality Risks from Various Sources.

    Who Drives the Quality Risk Analysis Bus?

    Eliminating the Appearance of Magic in the Quantification of Risk.

    Dealing with Variation in Ratings.

    Convincing Stakeholders to Invest Time and Accept Reality.

    Implement Improvements.

    3. Gaze into the Crystal Ball: Estimate the Work Ahead.

    An Estimation Process.

    Jamal Predicts the Future.

    Using Rules of Thumb for Rough Estimation.

    Effort, Duration, and Dependencies.



    Test Development.

    Test Environment Configuration.

    Test Execution.

    Moving on to Dollars and Cents.

    4. It's Not What It Costs, It's What It Saves: Budget and Return on Investment.

    Analyzing Return on Investment for Testing.

    Bugs That Get Fixed-or Prevented.

    Bugs That Don't Get Fixed-but Are Known.

    Tests That Mitigate Risks.

    Guiding the Project.

    A Final Note on Return on Investment Estimates.

    Jamal Prepares the Bill-and the Justification.

    Moving Forward, Then Looking Back.

    5. From Estimate to Baseline: Obtain Commitment to Realistic, Actionable, Truthful Estimates.

    Selling the Estimate.

    Jamal Makes His Case.

    Recognize a Good Estimation Process.

    Understand the Factors that Affect the Estimate.

    Paint a Realistic Picture of the Project's Future.

    Assign Responsibility for Action.

    Predict with Honesty.

    Estimate Quickly.

    Handle Challenges.

    What Kind of Investment Is Testing, Really?

    It Might Look Like Magic-But It's Only an Estimate.

    The Nine-Month Rule.

    Be Careful with Nonstakeholders.

    When Selling Fails.

    Safety- or Mission-Critical Systems.

    Implement Improvements.

    6. Gain and Communicate Insights: Plan the Test Effort.

    A Test Planning Process.

    Jamal Has a Plan.

    Beyond the Process: Key Considerations for the Test Plan.

    Selecting Strategies for Testing.

    A Special Case: Regression Risk Management Strategies.

    Understanding Test Environment Execution Resources.

    Managing Test Systems and Environment Configurations.

    Test Subproject Risks.

    Getting Everyone On Board.

    7. From Suggestion to Commitment: Assemble Stakeholder Support for Good Test Plans.

    Jamal Makes His Pitch.

    Recognize a Good Test Planning Process.

    Establish Clear Criteria for Phases.

    Establish Consensus, Common Expectations, and Commitment.

    Complete at a Reasonable Time.

    Promote Reasonable Flexibility and Creativity.

    Produce Appropriate Documentation.

    Provide Opportunities to Catch Errors.

    Exploit Synergies with the Project, Development, Build, and Integration Plans.

    Handle Challenges.

    Transcending the Template.

    Planning for Outsourcing.

    When Key Players Don't Support the Test Plan.

    Obeying the Law and Pleasing Regulators.

    Implement Improvements.


    8. Bring on the Great Testers: The How and the Who of Hiring.

    A Team Building Process.

    Variations on the Team Building Process.

    The Sumatra Test Team Grows.

    Identifying and Managing the Critical Test Team Skills.

    Regarding Test Technicians.

    Test Team Staffing Variations: Temporary Assignment, Rotation, Farm Team, and Backwater.

    Education, Training, Certification, and the Professionalization of Testing.

    Attitude Matters Too.

    Beyond the Obvious Interview Questions.

    From Adding People to Growing Skills.

    9. Grow Excellent Test Teams: Skills, Attitudes, and Career Paths.

    Jamal and Lin-Tsu Discuss Skills Growth.

    Using Skills Assessment as a Career Growth and Team Management Tool.

    Recognize a Good Team Building Process.

    Operate on a Win-Win Philosophy.

    Accurately Define the Job.

    Provide the Employee with a Long-Term Career Path.

    Give Employees Appropriate and Fair Salaries.

    Retain Employees with Sufficient Education, Experience, and Skill.

    Allow Appropriate Specialization.

    Handle Challenges.

    What Do Testers Need to Know-and What Can You Get?

    White Elephants.

    Using Contractors and Consultants.

    Implement Improvements.

    10. Archimedes' Bathtub: Design and Implement Test Systems.

    A Test System Design and Implementation Process.

    Emma Creates Stress on the Job.

    Three Crucial Considerations.

    Test Oracles.

    Appropriate Documentation.

    Combinatorial Explosions.

    Shifting to the Management Picture.

    11. Fill the Bathtub: Test System Coverage and Quality.

    A Brief Survey of Test Coverage Analysis Techniques.

    Jamal Assesses Coverage.

    Recognize a Good Test System Design and Implementation Process.

    Provides Capable Tools for Assessment of Quality.

    Efficiently Implements, Uses, Reuses, and Maintains the Test System.

    Selects the Right Techniques.

    Prevents Bugs.

    Handle Challenges.

    How Many Conditions in a Test Case?

    Retesting Third-Party Components.

    Vaguely Defined Requirements, Usage Profiles, and Environments.

    Balancing Coverage, Schedule, and Budget.

    Implement Improvements.


    12. An Essential Handoff: Manage Test Releases.

    A Test Release Process.

    The Big Build Hits the Test Lab.

    Recognize a Good Test Release Process.

    Provide Installable, Improved, and Stable Test Releases.

    Produce Predictable, Timely, and Regular Test Releases.

    Use Defined, Customer-Like Install and Uninstall Processes.

    Give the Test Release a Name That It Knows.

    Build from Carefully Selected, Coordinated, and Documented Content.

    Handle Challenges.

    Who Owns the Test Release Processes?

    Complexity of the System Under Test-and Beyond.

    Incomplete and Unrealistic System and Acceptance Test Releases.

    Implement Improvements.

    13. Assess Quality: Execute Test Cases.

    A Test Execution Process.

    Test Tracking Worksheets.

    The Attack on the Big Build.

    Recognize a Good Test Execution Process.

    Find the Scary Stuff First.

    Produce, Gather, and Disseminate Valuable Information.

    Correctly Interpret Test Results.

    Exhibit the Proper Attitudes.

    Proceed Efficiently.

    Handle Challenges.

    Dealing with Extended Shifts and Outsourced Testing.

    Accommodating Holidays, Cultures, and Vacations.

    Capturing the History on the Ground.

    Implement Improvements.


    14. Where Quality Falls Short: Report Bugs.

    A Bug Reporting Process.

    A Big Bug in the Big Build.

    Beyond the Failure Description.

    Recognize a Good Bug Reporting Process.

    Effectively Communicate Useful Information.

    Describe One Symptom per Report and File One Report per Symptom.

    Draw a Clear Boundary Between Testing and Debugging.

    Handle Challenges.

    Separating Bugs, Features, and "So What".

    Handling Bugs That Got Fixed by Accident and Irreproducible Symptoms.

    Keeping Noise Out of the Signal.

    Building Trust and Credibility with Programmers.

    Selecting the Right Bug Tracking Tool.

    Implement Improvements.

    15. Illuminate the Fourth Element: Report the Test Results.

    A Test Results Reporting Process.

    Jamal Defines a Dashboard-and Reports on the Big Build's Test Results.

    Recognize a Good Test Results Reporting Process.

    Deliver Useful, Credible, and Timely Information.

    Relate Test Status to Project Progress-and Influence Future Progress.

    Show Trends and Destinations in Quality over Time.

    Use Appropriate Mechanisms and Documents.

    Tune the Communication to Each Listener.

    Handle Challenges.

    Handling the Presentation.

    The Bearer of Bad News.

    An Ever-Changing Complex Story.

    Maintaining a Passion for Quality While Giving a Dispassionate Presentation.

    Implement Improvements.

    16. Leverage Opportunities to Learn: Manage Change and Its Effects on Testing.

    A Change Management Process.

    Jamal Makes a Case-and Accommodates.

    An Interconnected Process.

    Recognize a Good Change Management Process.

    Select the Right Changes in the Right Order.

    Balance Considerations of Features, Schedule, Budget, and Quality.

    Handle Challenges.

    The Complex Nature of Change Effects.

    The Ripple Effects of Change.

    Becoming a Roadblock-or Becoming Perceived as a Roadblock.

    Implement Improvements.

    17. Return to the Big Picture: Perfect the Testing Process.

    Recognize a Good Test Process.

    Provide Valuable, Economical Services.

    Pervade the Project.

    Use a Mature Process.

    Apply the Appropriate Level of Formality and Standardization.

    Employ Appropriate Strategies.

    Successfully Introduce Tools.

    Enjoy Continuous Learning and Inspiration.

    Handle Challenges.

    How Can We Measure What We Can't Define?

    Immature Development or Maintenance Processes.

    Unrealistic Expectations.


    "Addicted to Test".

    Incremental Process Improvement.

    Fine-Tune the Big Processes.

    Identify and Remove the Big Process Obstacles.

    Implement Improvements.

    Jamal Brown Looks Back-and Looks Forward.




    I’ve spent most of my twenty years in the software and hardware business inthe arena of testing. For the first few years as a test practitioner, I struggled tokeep my head above water. Ultimately, I mastered some basic tools and techniques.

    As I learned more about testing, I started to notice certain common themes.Some of these themes had to do with events—good and bad—that happenedover and over again on software, hardware, and system projects. For some of theseevents, I found that some teams could create order in their projects. These teamshandled these common events better than the teams that bounced from one crisis tothe next, reacting constantly, immersed in chaos. The successful teams had goodprocesses.

    Some of these successful project teams implemented writtenprocesses, while others accumulated “institutional knowledge” in theirwise—and sometimes prematurely gray—heads. While I have nothing againsta shared company culture, it’s hard to pass along the processes you’velearned unless you write them down-whether formally or informally, as checklists.This book takes the informal road. I describe twelve specific test processes, usingchecklists.Each process is critical to test team success.

    I describe theseprocesses in chronological order. First we plan the test activities. Next we prepareto test. After that, we perform the tests. Finally, we perfect the system under testand the testing activities themselves.

    Many other books have covered thetopics of preparing and performing tests in great detail. My experience is that, astesters, we generally do a good job in these areas. So, instead of rehashing what wealready know, I focus on opportunities for improvement. I devote eleven of theseventeen chapters to the topics of planning and perfecting. By far, these are theareas where we as testers have the most difficulty. This is especially true forcomplex and critical projects.

    Where will this book take you? During theearly colonization of the American continent in the 1540s, Francisco Vasquez deCoronado searched the deserts of present-day Arizona and New Mexico for the SevenCities of Cbola, including El Dorado, a city whose streets were supposedly pavedwith gold. Juan Ponce De Leon searched for a Fountain of Youth. In 1911, one of thefirst management consultants, Frederick Winslow Taylor, wrote a book called ThePrinciples of Scientific Management. Taylor espoused the idea of the one bestway—the perfect process—for each activity on an assembly line or in anyother industrial enterprise. But none of these three men found streets of gold, lifewithout death, or perfect processes.

    This book isn’t about Quixoticquests. There are no streets of gold that will make us effortlessly rich. Wecan’t side-step our human limitations. I don’t have infallibleprocesses. As Frederick Brooks wrote in the Mythical Man-Month, SecondEdition, we don’t have any silver bullets to kill our system projectmonsters, including the ones that live in quality and testing. That said, I havefound many ways for testers to deliver valuable information and services to theproject team, and each of these ways has its strong points and its weaknesses.

    The processes in this book might differ from what you’re doing now. In somecases you’ll decide, based on the success of your current processes, thatyou’re doing a fine job already. In some cases, though, you may want toimplement improvements. I’ll discuss specific ways to do that, but two themesapply to process change throughout the book.

    First, only change what’sbroken when changing it will help. Process change for its own sake, or processchange to perfect an already-good process, often doesn’t help the test team orthe organization. Indeed, such efforts can prove a dangerous distraction fromwhat’s truly important.

    Second, change is often easiest when done insteps wherever possible. Change should be made as painless as possible. All theprocesses in the book were developed through incremental change as I realized that abetter way of doing things would significantly increase the value my team could add,and fine-tuned my processes to achieve that.

    The processes in this bookaren’t pie-in-the-sky theory, but rather grew out of my experiences on theground as a practicing tester, test lead, and test manager. Your experiences andyour challenges will differ from mine, so be sure to adapt my processes—orcompletely reinvent your own—rather than trying to put a saddle on a cow.Following good processes can liberate you from the rote aspects of certain tasks,allowing you to focus on the fun, the fascinating, and the creative. When theprocesses you’ve adopted no longer solve the critical problems, when they needto evolve as your situations change, when they get in the way, then it’s timeto rethink how you do what you do. The processes I discuss here are lightweightchecklists (things I want to remember to do), not bureaucratic regulations (things Ihave to do because someone told me to).

    I hope that this book will start youthinking about the following questions: How do we do our testing jobs every day andon every test project as best as we possibly can? How do we de-institutionalize ourknowledge of how we do what we do? Even though we have varied experiences, is therea commonality of practices that we can share for critical testing processes thatdetermine our success? This book will give you a compendium of proven testingprocesses to help jump-start the most critical testing process of all, the thinkingprocess.



    Download the Index file related to this title.


    Submit Errata

    More Information

    InformIT Promotional Mailings & Special Offers

    I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.


    Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

    This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

    Collection and Use of Information

    To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

    Questions and Inquiries

    For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

    Online Store

    For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


    Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

    Contests and Drawings

    Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


    If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

    Service Announcements

    On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

    Customer Service

    We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

    Other Collection and Use of Information

    Application and System Logs

    Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

    Web Analytics

    Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

    Cookies and Related Technologies

    This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

    Do Not Track

    This site currently does not respond to Do Not Track signals.


    Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


    This site is not directed to children under the age of 13.


    Pearson may send or direct marketing communications to users, provided that

    • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
    • Such marketing is consistent with applicable law and Pearson's legal obligations.
    • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
    • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

    Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

    Correcting/Updating Personal Information

    If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


    Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

    Sale of Personal Information

    Pearson does not rent or sell personal information in exchange for any payment of money.

    While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

    Supplemental Privacy Statement for California Residents

    California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

    Sharing and Disclosure

    Pearson may disclose personal information, as follows:

    • As required by law.
    • With the consent of the individual (or their parent, if the individual is a minor)
    • In response to a subpoena, court order or legal process, to the extent permitted or required by law
    • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
    • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
    • To investigate or address actual or suspected fraud or other illegal activities
    • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
    • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
    • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


    This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

    Requests and Contact

    Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

    Changes to this Privacy Notice

    We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

    Last Update: November 17, 2020