Table of Contents
I. WHAT IS AUTOMATED TESTING? 1. The Birth and Evolution of Automated Testing.
Background on Software Testing.
The Automated Test Life-Cycle Methodology (ATLM).
Decision to Automate Test.
Test Tool Acquisition.
Automated Testing Introduction Phase.
Test Planning, Design, and Development.
Execution and Management of Tests.
Test Program Review and Assessment.
ATLMs Role in the Software Testing Universe.
ATLM Relationship to System Development Life Cycle.
Test Maturity Model (TMM)Augmented by Automated SoftwareTesting Maturity.
Test Automation Development.
Software Testing Careers. 2. Decision to Automate Test.
Overcoming False Expectations for Automated Testing.
Automatic Test Plan Generation.
Test Tool Fits All.
Immediate Test Effort Reduction.
Immediate Schedule Reduction.
Tool Ease of Use.
Universal Application of Test Automation.
One Hundred Percent Test Coverage.
Benefits of Automated Testing.
Production of a Reliable System.
Improvement of the Quality of the Test Effort.
Reduction of Test Effort and Minimization of Schedule.
Acquiring Management Support.
Test Tool Proposal. 3. Automated Test Tool Selection and Evaluation.
Organizations Systems Engineering Environment.
Third-Party Input from Management, Staff, and Customers andUsers.
Tool Criteria Reflecting the Systems Engineering Environment.
Level of Software Quality.
Help Desk Problem Reports.
Types of Tests.
Long-Term Investment Considerations.
Test Tool Process.
Tools That Support the Testing Life Cycle.
Business Analysis Phase Tools.
Requirements Definition Phase Tools.
Tools for the Analysis and Design Phase.
Programming Phase Tools.
Other Testing Life-Cycle Support Tools.
Testing Phase Tools.
Test Tool Research.
Evaluation Domain Definition.
Hands-On Tool Evaluation.
II. INTRODUCTION OF AUTOMATED TESTING TO A PROJECT. 4. Automated Testing Introduction Process.
Test Process Analysis.
Goals and Objectives of Testing.
Case Study: Test Objectives and Strategies.
Test Tool Consideration.
Review of Project-Specific System Requirements.
Review of Project Schedule.
Test Tool Compatibility Check.
Demonstration of the Tool to the Project Team.
Test Tool Support Profile.
Review of Training Requirements 5. Test Team Management.
Organizational Structure of a Test Team.
Stovepipe Test Team.
Centralized Test Team.
IV&V Test Team.
Systems Methodology and Test Team.
Test Team Summary.
Test Program Tasks.
Test Effort Sizing.
Test Team Sizing Methods: An Overview.
Development Ratio Method.
Test Procedure Method.
Task Planning Method.
Test Effort Sizing Factors.
Test Engineer Recruiting.
Test Engineer Qualities.
Test Team Composition.
Locating Test Engineers.
Test Engineer Interviews.
Distinguishing the Best Candidate.
Roles and Responsibilities.
III. TEST PLANNING AND PREPARATION. 6. Test Planning: Smart Application of Testing.
Test Planning Activities.
Test Program Scope.
Test Goals, Objectives, and Strategies.
Test Program Parameters.
Test Requirements Definition.
Test Requirements Management.
Requirements Management Tools.
Assessing the Test Requirements Risk.
Prioritization of Tests.
Requirements Traceability Matrix.
Test Program Events, Activities, and Documentation.
The Test Environment.
Test Environment Preparations.
Test Environment Integration and Setup.
The Test Plan.
Test Completion/Acceptance Criteria.
Sample Test Plan. 7. Test Analysis and Design.
Test Requirements Analysis.
Development-Level Test Analysis (Structural Approach).
System-Level Test Analysis (Behavioral Approach).
Test Program Design.
Test Program Design Models.
White-Box Techniques (Development-Level Tests).
Black-Box Techniques (System-Level Tests).
Test Design Documentation.
Test Procedure Design.
Test Procedure Definition.
Automated Versus Manual Test Analysis.
Automated Test Design Standards.
Case Study: Naming Conventions.
Manual Test Design Guidelines.
Detailed Test Design.
Test Data Requirements. 8. Test Development.
Test Development Architecture.
Environment Readiness Checks.
Automation Reuse Analysis.
Test Procedure Development/Execution Schedule.
Explanation of the Sample Modularity- Relationship Matrix.
Calibration of the Test Tool.
Compatibility Work-Around Solutions.
Case Study: Incompatibility Work-Around Solution.
Manual Execution of Test Procedures.
Test Procedure InspectionsPeer Reviews.
Test Procedure Configuration Management.
Test Development Guidelines.
Reusable Test Procedures.
Case Study: Navigation Using Tabs or Mouse Clicks.
Case Study: Testing Bitmaps Using a Capture/Playback Tool.
Maintainable Test Procedures.
Case Study: Automating Documentation.
Case Study: Automated Random Testing.
Table-Driven Test Automation.
PC Environment Automated Setup Script.
Automated Recording Options.
Verifying GUI Standards.
Case Study: Smoke Test Application.
Help Function Verification Script.
Timed Message Boxes Function.
Advanced Math Functions.
IV. TEST EXECUTION AND REVIEW. 9. Test Execution.
Executing/Evaluating Test Phases.
Unit Test Execution and Evaluation.
Integration Test Execution and Evaluation.
System Test Execution and Evaluation.
Test Results Analysis of Regression Tests.
User Acceptance Test Execution and Evaluation.
Defect Tracking and New Build Process.
Defect Life-Cycle Model.
Test Program Status Tracking.
Earned Value Management System.
Case Study: System Test Status Tracking.
Test Metrics Collection and Analysis. 10. Test Program Review and Assessment.
Test Program Lessons LearnedCorrective Actions andImprovement Activity.
Test Program Return on Investment.
Case Study: TestProgram Return on Investment.
Case Study: Quantify Tool Return on Investment.
V. APPENDIXES. A. How to Test Requirements.
Requirements Testing Approach.
The Quality Gateway.
Make the Requirement Measurable.
Coherency and Consistency.
Requirement or Solution?
Order in a Disorderly World.
References. B. Tools That Support the Automated Testing Life Cycle.
Business Analysis Phase.
Business Modeling Tools.
Configuration Management Tools.
Defect Tracking Tools.
Technical Review Management.
Requirements Definition Phase.
Requirements Management Tools.
Use Case Generators.
Analysis and Design Phase.
Visual Modeling Tools.
Structure Charts, Flowcharts, and Sequence Diagrams.
Test Procedure Generators.
Memory Leak and Runtime Error Detection Tools.
Static and Dynamic Analyzers.
Unit Testing Tools.
Code (Test) Coverage Analyzers and Code Instrumentors.
Usability Measurement Tools.
Testing Support Tools.
Test Data Generators.
File Comparison Tools.
Test Management Tools.
Network Testing Tools.
GUI Application Testing Tools.
Load/Performance Testing Tools.
Web Testing Tools.
Year 24710 Testing Tools.
Other Test Tool Vendors. C. Test Engineer Development.
Technical Skills Stage.
Test Process Stage.
Team Effort Stage.
Technical Stewardship Stage.
Test/Project Management Stage.
Business/Product Management Stage. D. Sample Test Plan.
Roles and Responsibilities.
Project Roles and Responsibilities.
Test Task Structure.
Test Team Resources.
Test Environment Configuration.
Test Program Reporting.
Test Program Metrics.
Detailed Test Schedule. Appendixes. D.A Test Procedure Development Guidelines. D.B Test Verification Summary and Matrix. D.C Test Procedures and Test Scripts E. Best Practices.
Test Tool Compatibility Checks.
Test Tool Upgrades.
Baselined System Setup and Configuration.
Software Installations in the Test Environment Baseline.
Overall Test Program Objectives.
Keep Automation Simple.
Test Procedure Design and Development Standards.
Automated Versus Manual Test Analysis.
Test Team Communication with Other Teams.
Defect Documentation and Reporting.
Automated Test Advocates and Experts.
Test Team Assignments.
User Group Participation.
Test Tool Improvement Suggestions.
Become a Beta Testing Site.
Specialty Topic Experts. 0201432870T04062001
Preface Automated Software Testing
addresses the challenge for todays software professionals who are faced with real schedule deadlines and need to introduce, manage, and perform automated testing on a project. The book addresses automated testing within a client-server or Web environment.
The focus of this book is the pragmatic concerns and information needed by the software test engineer/manager
who faces the necessity of performing testing more thoroughly and quickly. By the same token, these same concerns may apply to the software developer
who has the responsibility for development testing (that is, unit and integration testing) and, on some projects, system testing. The book also represents an informative guide that bolsters the ability of the quality assurance engineer
to perform quality reviews on test design, test procedures, and the results of test activities.
The software project manager
, who is responsible for the overall development effort, may also find this book useful. The text provides the project manager with guidelines concerning the goals and objectives of the testing effort and the decision about whether to automate testing. It also offers guidance on introducing automated testing on a project and outlines the processes for performing test planning, design, development, execution, and evaluation.
The authors have worked intimately with a number of automated testing professionals around the world, who were generous enough to share their problems and concerns. One primary concern voiced by these test engineers related to the fact that the test industry does not have the kind of structured methodologies that developers have traditionally enjoyed. Similarly, project managers, test managers, and test engineers may not be familiar with the kinds of approaches that are required to perform automated testing as opposed to the traditional test approach.
Clearly, the emphasis on automated testing represents a paradigm change for the software industry. This change does not simply involve the application of tools and the performance of test automation. Rather, it is pervasive across the entire test life cycle and the system development life cycle. The approach taken by project managers, test managers, software engineers, and test engineers is altered as a result. For software professionals to successfully make the leap to automated testing, structured approaches to testing must be embraced. Automated Software Testing
is revolutionary in that it promulgates a new structured, building-block approach to the entire test life cycle, while also providing relevant test automation and associated test management guidance needed by industry test professionals. Automated Testing
Software project managers and software developers building todays applications face the challenge of doing so within an ever-shrinking schedule and with minimal resources. As part of their attempt to do more with less, organizations want to test software adequately, but as quickly and thoroughly as possible. To accomplish this goal, organizations are turning to automated testing.
Faced with this reality and realizing that many tests cannot be executed manually, such as simulating 1,000 virtual users for volume testing, software professionals are introducing automated testing to their projects. While needing to introduce automated testing, software professionals may not know whats involved in introducing an automated test tool to a software project, and they may be unfamiliar with the breadth of application that automated test tools have today. Automated Software Testing
provides guidance in these areas.
The growth of automated test capability has stemmed in large part from the growing popularity of rapid application development (RAD), a software development methodology that focuses on minimizing the development schedule while providing frequent, incremental software builds. The objective of RAD is to engage the user early and throughout design and development of each build so as to refine the software, thereby ensuring that it more closely reflects the needs and preferences of the user. In this environment of continual changes and additions to the software through each software build, where requirements are encouraged to evolve, software testing takes on an iterative nature itself. Each new build is accompanied by a considerable number of new tests as well as rework to existing test scripts, just as there is rework on previously released software modules. Given the continual changes and additions to software applications, automated software testing becomes an important control mechanism to ensure accuracy and stability of the software through each build.
As noted above, a primary goal of RAD is to shorten the overall development schedule, by addressing the riskiest aspects of development in early builds. As a result, test activities are undertaken at the start of the initial RAD cycle and through each subsequent RAD cycle as well. As noted in Part III of this book, test design and development represent a complex undertaking. A test effort, in fact, may be as time-consuming as the effort required to develop the software application. When the project involves the integration of commercial off-the-shelf (COTS) products, for example, the test effort may even require more resources than software development. Instances where the test team does not participate in software specification or when test is not initiated soon enough pose a risk to the project. In these situations, potential outcomes include an incomplete software test effort, an insufficient test schedule, and an unplanned extension to the development schedule to accommodate testing.
Much of the test effort required on a project now needs to be supported by automated test tools. Manual testing is labor-intensive and error-prone, and it does not support the same kind of quality checks that are possible with an automated test tool. The introduction of an automated test tool can replace mundane manual test activities with a more efficient and repeatable automated test environment, which itself improves test engineer morale and retention.
Although some automated test tools began as capture and playback tools, the functionality and capabilities of automated test tool suites have been expanding. Automated test capabilities for software products include testing of the graphical user interface, requirements compliance, load performance, code coverage, Web interface, network communications, memory leakage, and more. New capabilities continue to be added to keep pace with the growing demand for test support. ATLM
This book concentrates on the concerns of the software test professional within the framework of an Automated Test Life-cycle Methodology (ATLM). ATLM is a structured methodology geared toward ensuring successful implementation of automated testing. The ATLM approach mirrors the benefits of modern, rapid application development efforts, where such efforts engage the user early in the development cycle. The end user of the software product is actively involved throughout analysis, design, development, and test of each software build, which is augmented in an incremental fashion.
The ATLM incorporates a multistage process consisting of six components. It supports the detailed and interrelated activities that are required to decide whether to acquire an automated testing tool. The methodology takes into account the process of introducing and optimizing an automated test tool and addresses test planning, analysis, design, development, execution, and management. The scope of the test program is outlined within the test plan, as a top-level description of test approach and implementation. The scope is further refined through the definition of test goals, objectives and strategies, and test requirements. Similar to software application development, test requirements are specified before test design is constructed. Likewise, the test program must be mapped out and consciously designed to ensure that the most efficient and effective tests for the target application are performed. Test design is developed through graphical portrayals of the test effort, so as to give project and test personnel a mental framework on the boundary and scope of the test program. Test Training
The evolution of automated testing has given birth to new career opportunities for software engineers. In fact, while the demand for automated software test professionals has exploded, community colleges and universities have not produced a reciprocal response to help support industry demand.
Some universities that are typically more proactive in responding to changes in the software industry have already implemented software test and quality assurance courses. For example, George Mason University (GMU) provides software test and quality assurance courses (see the GMU Web site at http://www.isse.gmu.edu/
for more information). Kansas State University (KSU) offers students several courses on the subject of software test and quality assurance and other courses that touch on the subject (see the KSU Web site at http://www.ksu.edu
Purdue University offers two undergraduate courses in software engineering that cover software testing and reliability. Purdue also has established a software engineering research center (http://serc.vuoregon.edu/serc/
), in which faculty from eight universities and representatives from eleven companies participate. Among other areas this center supports research projects. For more information, see the Purdue University Web site at http://www.purdue.edu
The North Seattle Community College has established one of the most progressive testing curricula in the country. The college offers three levels of software testing courses (introduction, automation, and leadership) and one- and two-year software testing programs. For more information about these courses and programs, visit the their Web site at http://nsccux.sccd.ctc.edu/
. Information concerning additional university and industry training resources is available on the authors Web site at http://www.autotestco.com/
The Automated Software Testing
textbook is intended to help in classroom instruction on software testing that uses modern automated test tool capabilities. The book provides students with an introduction to the application and importance of software test, and it describes the different kinds of automated tests that can be performed. Instruction continues with the definition of the test approach, the roles and responsibilities of the test team, test planning, test design, test script development, test execution, defect tracking, and test progress reporting. About the Authors Automated Software Testing
was developed by three software industry professionals. Elfriede Dustin
has performed as a computer systems analyst/programmer developing software applications and utilities, process and data modeling using CASE tools, and system design simulation models. She has experience supporting a variety of system application development projects, including health, financial, logistic, and enterprise information management systems. In addition, Elfriede has been responsible for implementing the entire development life cycle, from requirement analysis, to design, to development, to automated software testing. She has been a test manager and a lead consultant guiding the implementation of automated testing on many projects. Because of her automated test expertise, she has been sought out to help modify the capabilities of commercial test tool products, where her use and feedback on test products has proved invaluable. Jeff Rashka
has managed a multitude of significant information system and systems integration projects. System applications on which he has served as manager include worldwide transportation asset management, enterprise information management, financial management, bar-coded inventory management, and shipboard information systems. Jeff also has process improvement management experience in implementing the guidelines contained within the Software Engineering Institutes Capability Maturity Model (CMM). John Paul
has worked as a senior programmer/analyst on financial and budgeting systems as well as a host of other information systems. His software development leadership responsibilities have included system analysis and design, application prototyping, and application development using a number of different methodologies and programming techniques. His software development responsibilities have included application testing using automated test tools. John has also assumed a lead role in the performance of year 2000 compliance testing.
The authors have applied their collective knowledge of software engineering, automated testing, and management to develop a book that addresses the pragmatic concerns and information needed by the software test engineer and manager. Automated Software Testing
is designed to be a usefuland practicalguide for software engineers and software project managers who are responsible for software test activities. Book Style and Organization
This books organization correlates with the phases, tasks, and steps of the ATLM. The sequence of the book is fashioned in a purposeful way. It addresses the reader as if he or she had just been handed a note giving that individual the responsibility for automated software testing on a project. A first question might be, "What exactly is automated testing, and why do I need it?" Part I of the book answers this question and provides other fundamental instruction allowing the would-be test to approach the new responsibility with confidence. The reader is then guided through the decision to automate testing as well as automated test tool selection and evaluation.
After receiving this fundamental instruction, the test engineer would have several more questions: "What is involved in setting up the tool?" "How do I get the test team in place?" "What early test planning is required?" Part II answers these questions. Specifically, the process for introducing an automated test tool is outlined and guidelines for structuring the test team are provided.
Part III focuses on automated test planning, analysis, design, and test automation (programming). This section of the book addresses test design techniques, which are comparable to the structured software design techniques introduced over the past 20 years. Specifically, it highlights the discipline required in the design of test automation. The goal is to give useful information on test design and test case development, so that the test engineer doesnt have to discover (by trial and error) how to put together a good test design and set of test procedures.
Part IV helps the reader address several additional questions: "Whats involved in the performance of test?" "How do I manage my test schedule?" "How do I document and track defects?" This section provides guidelines pertaining to test execution, defect tracking, and test program status tracking. A set of best practices for the development and execution of automated test procedures is provided to assist test professionals in executing test activities in an efficient manner.
Overall, the book seeks to allay the concerns of the software test professional within the framework of the ATLM. ATLM is a structured methodology, and one that is geared toward ensuring successful implementation of automated testing. Readers with questions and comments may contact the authors via their home page at http://www.autotestco.com/
. This Web site also provides more information on automated software testing and resources that are available to support automated software testing programs.