- Overview
- State of the Practice
- Impact on Testing
- Solutions to the Challenge
- Solution Impediments
- Guidelines for Success
- Plan of Action
This chapter is from the book
This chapter is from the book
This chapter is from the book
State of the Practice
Joe and Mary are members of the testing organization at Fantastic Foods, a large corporation managing hundreds of grocery stores. Joe is on a major project to develop a new data warehousing system, and Mary is working on an independent test team to test changes made to the company’s twenty-year-old inventory system.
One day at lunch, Joe and Mary discover that although they are working on two totally separate and unrelated projects, they share a common problem: too many things to test and not enough time to test them. “It just seems like the project schedule was based on the deadline—as if we could fit in all the work,” observed Joe. “This project is so large and complex, I don’t know how we’ll ever get any level of confidence that the system really works. My management just says to test everything we have time to test, but at the same time, they say that the department’s reputation rests on this project. It’s like we’re in an impossible situation, yet nobody wants to admit it,” Joe said.
Mary agreed, “Yeah, I don’t know how much longer I can stand getting the same programs to test over and over. What’s worse is that the defects seem to be getting worse instead of better. I got one program yesterday at ten A.M. that had to be tested by three P.M. The regular test takes over twelve hours to run! Tell me, how do I keep doing this without sacrificing the quality of the test?”
The problems that Joe and Mary face in this scenario are
They expect themselves to test everything.
Although everyone at Fantastic Foods understands that the systems under test are complex, no one seems to transfer this understanding to testing. This inconsistent expectation is even held by the testers. Because everything is fair game for testing, there is no effort to define or control the scope of testing.
Their management expects them to test everything.
To their credit, the managers understand the need for quality control (testing) on the system projects at Fantastic Foods. However, increased awareness of the dynamics of testing large and complex systems is needed. In this case, management should be leading the effort in defining a risk-based process for specifying and controlling the scope of testing. Until management understands that it is impossible to test everything, there will likely be a false assertion of blame on the testers if a defect is found by a customer or user, usually in the form of “I thought you guys were supposed to be testing this system. How did you miss this problem?”
There is not enough time to test everything.
If you examine the project schedule and the delivery process for changes to the legacy system, you will see that a set amount of time is defined for testing. We call this allotted time “the box.” The challenge that testers have is to fit their testing into the box, and sometimes the box gets smaller if other parts of the project fall behind schedule. Realizing what is really going on in terms of time helps people to set more realistic expectations of what can be accomplished.
Let’s be more specific as to what is or is not reasonable in terms of test objectives. What is not reasonable is to expect to test every random combination of events or paths in an average software module. Even if you could completely test all combinations in one module, the complexity explodes when a second or third module is introduced. What is reasonable is to use an approach like requirements-based testing to pinpoint the test cases you need to perform. This approach can reduce the number of test cases in some instances from hundreds of thousands to less than one hundred.
The testing estimates are not based on any kind of measurable criteria.
Not only do the estimates fail to define or to control the scope of testing, the estimates for testing are not based on anything measurable. This reinforces the ineffective technique of fitting testing into the allotted time—regardless of whether or not the testing is effective. The reason that estimates are only best guesses stems from the fact that the measurable criteria are not available until later in the project. The good news is that there are ways to get measurable criteria early in the project.