- Testing Axioms
- Software Testing Is a Risk-Based Exercise
- Testing Can't Show That Bugs Don't Exist
- The More Bugs You Find, the More Bugs There Are
- The Pesticide Paradox
- Not All the Bugs You Find Will Be Fixed
- When a Bug's a Bug Is Difficult to Say
- Product Specifications Are Never Final
- Software Testers Aren't the Most Popular Members of a Project Team
- Software Testing Is a Disciplined Technical Profession
- About This Article
If you decide not to test every possible test scenario, you've chosen to take on risk. In the calculator example, what if you choose not to test that 1024+1024=2048? It's possible the programmer accidentally left in a bug for that situation. If you don't test it, a customer will still use it, and he or she will discover the bug. It'll be a costly bug, too, since it wasn't found until the software was in the customer's hands.
This may all sound pretty scary. You can't test everything, and if you don't, you will likely miss bugs. The product has to be released, so you will need to stop testing, but if you stop too soon, there will still be areas untested. What do you do?
One key concept that software testers need to learn is how to reduce the huge domain of possible tests into a manageable set, and how to make wise risk-based decisions on what's important to test and what's not.
Figure 2 shows the relationship between the amount of testing performed and the number of bugs found. If you attempt to test everything, the costs go up dramatically and the number of missed bugs declines to the point that it's no longer cost effective to continue. If you cut the testing short or make poor decisions of what to test, the costs are low but you'll miss a lot of bugs. The goal is to hit that optimal amount of testing so that you don't test too much or too little.
Every software project has an optimal test effort.