Software Testing In The Real World: Improving The Process

Description

Software Testing In The Real World provides the reader with a tool-box for effectively improving the software testing process. The book gives the practicing software engineer a menu of techniques with guidance on how to create a strategy for continuous, sustainable improvement within their organization--whatever its size or level of process maturity. Ed Kit addresses the most frequently asked questions about methodologies, tools, technology and organizational issues being posed in the testing community today. Pragmatic in its approach, the book confronts the problem of the relative immaturity of the software engineering discipline in most organizations with practical guidance on cost and risk, standards, planning testing tasks and testing tools. Test and Quality Assurance Specialists, Developers and Project Managers alike will benefit from the practical, proven techniques for improving testing as well as the specific "best of breed" software testing tools information.

Visit the Author's Home Page

Extras

Web Resources

Click below for Web Resources related to this title:
Author's Home Page

Sample Content

Table of Contents

PART I: Software testing process maturity
Chapter 1:
The six essentials of software testing
Essential 1: The quality of the test process determines the success of the test effort
Essential 2: Prevent defect migration by using early life-cycle testing techniques
Essential 3: The time for software testing tools is now
Essential 4: A real person must take responsibility for improving the testing process
Essential 5: Testing is a professional discipline requiring trained, skilled people
Essential 6: Cultivate a positive team attitude of creative destruction
Chapter 2:
The state of the art and the state of the practice
The short eventful history of a very new discipline
Where exactly are we now?
How should testing be positioned?
Chapter 3
The clean-sheet approach to getting started
PART II: The framework for test process improvement
Chapter 4
Establishing a practical perspective
What are we aiming for?
All you ever wanted to know about errors
So what is testing really?
Some definitions
Good testers have a testing attitude
How testers do itWhat can we do now?
Chapter 5
Critical choices: what, when, and how to test
Risk and risk management
Start testing early
Basic forms of the testing process - verification and validation
Testing, the development cycle, and the real world of contracts
Effective testing......and cost-effective testing
What can we do now?
Chapter 6
Critical disciplines: frameworks for testing
Planning
Software engineering maturity and the SEI
Configuration management
Standards
Formal documents
Testware
Measurement
Tools
What can we do now?
Part III: Testing methods
Chapter 7
Verification testing
Basic verification methods
Getting leverage on verification
Verifying documents at different phases
Getting the best from verification
Three critical success factors for implementing verification
Recommendations
Chapter 8
Validation testing
Validation overview
Validation methods
Validation activities
Recommendation strategy for validation testing
Chapter 9
Controlling validation costs
Minimizing the cost of performing tests
Minimizing the cost of maintaining the tests
Minimizing validation testware development costs
The testware library
Recommendations
Chapter 10
Testing tasks, deliverables, and chronology
Master test planning
Verification testing tasks and deliverables
Validation testing tasks and deliverables
A testing orphan - user manuals
Product release criteria
Summary of IEEE/ANSI test related documents
Life-cycle mapping of tasks and deliverables
References
Chapter 11
Software testing tools
Categorizing test tools
Tool acquisition
Chapter 12
Measurement
Measurement provides answers
Useful measures
Other interesting measures
Recommendations
Part IV: Managing test technology
Chapter 13
Organizational approaches to testing
Organizing and reorganizing testing
Structural design elements
Approaches to organizing the test function
Selecting the right approach
Chapter 14
Current practices, trends, challenges
GUIs: What's new here?
Usage testing
Tester-to-developer ratios
Software measures and practices benchmark study
Chapter 15
Getting sustainable gains in place
Getting gains to happen
Getting help
Follow-up
Appendix A Standards relevant to software engineering and testing
Appendix B Verification checklists
Appendix C Verification exercises
Appendix D Validation exercises (solutions)
Appendix E Bibliography
Appendix F Sources: conferences, journals, newsletters, DOD specifications
Appendix G Specific tools and tool selection
Appendix H Sample lists of improvements to be implemented
Index

Preface

Software testing.
It is exhilarating. It is exasperating.
It can be the best of jobs--or the worst.
It is fundamental to delivering quality software on time within budget.

This book is for anyone interested in improving their test process, including test specialists, product developers, managers, quality assurance specialists, maintainers, project managers, auditors, systems analysts, tool developers, and others whose jobs involve understanding software testing. It offers developers and other people whose work is not primarily in the testing area an opportunity to improve their own specialty through a better understanding of testing.

To understand testing and to improve testing practice, it is essential to see the software testing process in its broadest terms--as the means by which people, methodology, tools, measurement, and leadership are integrated to test a software product. More than 20 years in testing practice, teaching, and consulting have shown me the following key questions raised again and again by people wanting to improve their testing process:

Methodology questions
• What are the key testing methods, when do we use them, and how do we implement them?
• How do we prevent defect migration and improve defect finding capabilities?
• How can we understand and leverage the relevant test standards and terminology?
• How do we create meaningful test plans and other critical test deliverables?
• How do we identify and prioritize potential software testing process improvements?

Leadership questions
• How do we organize the software testing effort so it is measurable and controllable?
• Where can we find additional help regarding software testing?
• How can we ensure that improvements are sustainable in the medium and long term?
• How can we organize testing for the most effective relations between people?
• How can we generally maximize exhilaration and minimizeexasperation?

Tools and technology questions
• What are the most significant opportunities for automating the testing process?
• How can we use measurement to understand and control the software test process?
• How can we select test tools and vendors and implement an effective tools program?
• How can we use risk analysis techniques as a basis for decision making in testing?

My experience has shown that a balanced strategy addressing methodology, leadership, and technology is most effective for improving testing.

Most software development organizations suffer from immature software processes. Given this starting point, practitioners and managers naturally ask: "Yes, but what do I do now?". This book offers a tool-box for effectively improving the software testing process. A tool-box is not a single methodology to be followed from A to Z. It is a menu of techniques that can be taken individually or in combination to provide the means to formulate and reach realistic improvement objectives.

Part I is for orientation. It describes the six essentials of software testing, the history of testing, and a simple, practical approach to getting started with improving the testing process. It identifies software engineering in general and testing in particular as new disciplines grappling with the escalating demands of an environment expecting miracles.

Part II establishes practical goals from a "now we are here" position, and explains how these relate to formal and less formal definitions of testing and testing objectives, and to the current standards that are fundamental (and useful) to practitioners in their everyday work.

The basic forms of the testing process are explained, and the critical real-world issues of cost and risk are fully clarified and integrated in the decision making process of what, when and how to test. In this part we address the items that need to be in place before we dig more deeply into the testing activities of verification and validation. The issues of planning, software engineering maturity goals, configuration management, standards, and tools are explained and positioned.

Part III explains the basic verification and validation testing tasks, including planning and controlling testing costs, and making the best use of resources. The use of tools as a way of gaining leverage in the various testing activities is explained, together with clarification of tool categories, and practical guidance on the critical questions for tool acquisition. The appropriate and constructive use of measurement is explained, and guidelines given for the proper use of measurement results.

Part IV explains the organizational and human management issues that surround the test effort and which are key determinants of its success. It surveys some current best practice sources, and then refocuses on how to get sustainable improvements in place in the short and medium term.

Definitions, references, and examples are integrated within the main text of the book as far as possible. The appendices contain a variety of material that is too detailed for inclusion in the main text of the book, and which is essentially complementary. This includes sample checklists, exercises, planning documents, and sample working documents. There is further detail on official standards and specifications as well as useful help sources such as journals, tools, conferences, newsletters, user groups, and an annotated bibliography.

Acknowledgments

Life provides many teachers. Some say all of life is our teacher. Space prohibits acknowledging all who have positively influenced this book, but does allow acknowledgment of the following people due special appreciation.

Great appreciation is due my wife, Cathy Baldwin, who not only provided continuous moral support and endless editing of the book, but also provided encouragement during the early start-up years of Software Development Technologies. Special thanks to our families for their support over the years.

I am particularly grateful to David Gelperin and Bill Hetzel: to David for providing the opportunity to teach my first course on software testing nearly a decade ago, and for providing the ongoing friendship and support that kept me going; to Bill for encouraging me to write a book and forwriting the foreword for the book; and to both of them for providing the opportunity to be associated with Software Quality Engineering for many years.

Appreciation is due Chris Larson, who for 15 years has generously shared his friendship and wealth of software engineering experiences gained from over 25 years with Hewlett Packard, IBM, Tandem, and CDC, and who contributed enormously to the course on which this book is based.

Appreciation is also due Susannah Finzi for being an outstanding friend and editor, and for keeping the book appealing from a global perspective while patiently enduring seemingly endless rewrites without a single complaint.

Appreciation is due the reviewers who gave their time to providevaluable comments, especially Bill Hetzel, David Gelperin, Dot Graham, Marty Gipson, Vijay Sarathy, Ray O'Neal, Ellen Brodbine, Trevor Reeve, Cathy Baldwin, Chris Larson, Denise Leigh, and Paul Claydon.

Finally, appreciation is due to the students who attended the courses upon which this book is based and who in the process became my teachers and provided suggestions for improving the course, especially the participants from Hewlett Packard, Motorola, Microsoft, Apple Computer, US Army, IBM, NCR, Johnson & Johnson, Grumman Data Systems, Australia and New Zealand Bank, ADP, EDS, PeopleSoft, SCO, General Electric, Oracle, NEC, Lloyds Bank, USDA, NetFrame, Cadence, Sears, AT&T, Informix, British Aerospace, Sybase, Tandem, and Octel Communications.

Ed Kit
Cupertino, September 1995

Updates

Submit Errata