Home > Articles > Programming

  • Print
  • + Share This
This chapter is from the book

1.5 The Limitations of Testing

In spite of its critical nature, testing has a number of pitfalls that make it far less effective and efficient than it should be. Testing is relatively ineffective in the sense that a significant number of residual defects remain in a completed system when it is placed into operation. Testing is also relatively inefficient when you consider the large amount of effort, funding, and time that is currently spent to find defects.

According to Capers Jones, most types of testing find only about 35% of the software defects [Jones 2013]. This is consistent with the following, more detailed analysis of defect detection rates as a function of test type and test capabilities, as shown in Table 1.3 [McConnell 2004].

Table 1.3 Defect Detection Rate

Defect Detection Rates

Test Type

Lowest

Mode

Highest

Unit Test

15%

30%

50%

Component Test

20%

30%

35%

Integration Test

25%

35%

40%

System Test

25%

40%

55%

Regression Test

15%

25%

30%

Low-volume Beta Test

25%

35%

40%

High-volume Beta Test

60%

75%

85%

Source: McConnell 2004

As Table 1.4 shows, no single type of testing is very effective at uncovering defects, regardless of defect type. Even when all of these testing methods are used on an average project, they still only identify four out of five of the code defects.

Table 1.4 Defect Detection Rate

Static Verification

Project Defect Detection Rate

Worst

Average

Best

Desk Checking

23%

25%

27%

Static Analysis

0%

55%

55%

Inspection

0%

0%

93%

Static Subtotal

19%

64%

98%

Testing

Project Defect Detection Rate

Worst

Average

Best

Unit Test

28%

30%

32%

Function Test

31%

33%

35%

Regression Test

10%

12%

14%

Component Test

28%

30%

32%

Performance Test

6%

10%

14%

System Test

32%

34%

36%

Acceptance Test

13%

15%

17%

Testing Subtotal

72%

81%

87%

Cumulative Total

81.1%

95.6%

99.96%

Source: Jones 2013b

  • + Share This
  • 🔖 Save To Your Account