This chapter is from the book
This chapter is from the book
This chapter is from the book
4.6 Summary
Software quality metrics focus on the quality aspects of the product, process, and project. They can be grouped into three categories in accordance with the software life cycle: end-product quality metrics, in-process quality metrics, and maintenance quality metrics. This chapter gives several examples for each category, summarizes the metrics programs at Motorola, Hewlett-Packard, and IBM Rochester, and discusses data collection.
Product quality metrics
-
Mean time to failure
-
Defect density
-
Customer-reported problems
-
Customer satisfaction
In-process quality metrics
-
Phase-based defect removal pattern
-
Defect removal effectiveness
-
Defect density during formal machine testing
-
Defect arrival pattern during formal machine testing
To my knowledge, there is no correlation between the existence of a metrics program and the size of the organization, or between the effectiveness of using metrics in software development and team size. For small teams that are starting to implement a metrics program with minimum resources, I recommend the following:
Implement the defect arrival metric during the last phase of testing, augmented by a qualitative indicator of the number of critical problems (or show stoppers) and the nature of these problems. The latter is a subset of total defects. These indicators provide important information with regard to the quality of the software and its readiness to ship to customers. For example, if the defect arrival metric is trending down to a sufficiently low level and the number of show stoppers is small and becoming sparser as the test progresses, one can forecast a positive quality posture of the product in the field, and vice versa.
After the product is shipped, a natural metric is the number of defects coming in from the field over time. As defects are reported, the organization fixes these defects for their customers, normally in the form of individual fixes, fix packages, or frequent software upgrades, depending on the organization's maintenance strategy. The inevitable question is, how many outstanding defects need to be fixed; in other words, how big is the fix backlog. If the organization intends to compare its fix efficiency to the volume of defect arrivals, the backlog management index metric can be formed easily without additional data.
When a product size metric is available, the above metrics can be normalized to form other metrics such as defect density curves over time, during testing or when the product is in the field, and overall defect rate of the product, for a specific duration or for the maintenance life of the product. These normalized metrics can serve as baselines for other projects by age number of source statements (LOC) per function point) by language are as follows:
Maintenance quality metrics
-
Fix backlog
-
Backlog management index
-
Fix response time and fix responsiveness
-
Percent delinquent fixes
-
Defective fixes
With regard to in-process data, generally those at the back end (e.g., testing defects) are more reliable than those at the front end (e.g., design reviews and inspections). To improve data reliability, it is important to establish definitions and examples (e.g., what constitutes a defect during design reviews). Furthermore, validation must be an integral part of the data collection system and should be performed concurrently with software development and data collection.
|
Basic Assembly |
320 |
|
Macro Assembly |
213 |
|
C |
128 |
|
FORTRAN |
107 |
|
COBOL |
107 |
|
Pascal |
91 |
|
PL/I |
80 |
|
Ada83 |
71 |
|
C++ |
53 |
|
Ada95 |
49 |
|
Visual Basic |
32 |
|
Smalltalk |
21 |
|
SQL |
12 |
This discussion of a few simple and useful metrics as the beginning of a good metrics practice is from the quality perspective. For overall project management, it is well known that the most basic variables to measure are product size, effort (e.g., person-year), and development cycle time.
References
Albrecht, A. J., "Measuring Application Development Productivity," Proceedings of the Joint IBM/SHARE/GUIDE Application Development Symposium, October 1979, pp. 83–92.
Basili, V. R., and D. M. Weiss, "A Methodology for Collecting Valid Software Engineering Data," IEEE Transactions on Software Engineering, Vol. SE-10, 1984, pp. 728–738.
Boehm, B. W., Software Engineering Economics, Englewood Cliffs, N.J.: Prentice-Hall, 1981.
Conte, S. D., H. E. Dunsmore, and V. Y. Shen, Software Engineering Metrics and Models, Menlo Park, Calif.: Benjamin/Cummings, 1986.
Cusumano, M. A., "Objectives and Context of Software Measurement, Analysis and Control," Massachusetts Institute of Technology Sloan School of Management Working Paper 3471-92, October 1992.
Daskalantonakis, M. K., "A Practical View of Software Measurement and Implementation Experiences Within Motorola," IEEE Transactions on Software Engineering, Vol. SE-18, 1992, pp. 998–1010.
DeMarco, T., Controlling Software Projects, New York: Yourdon Press, 1982.
Fenton, N. E., and S. L. Pfleeger, Software Metrics: A Rigorous Approach, 2nd ed., Boston: International Thomson Computer Press, 1997.
Grady, R. B., and D. L. Caswell, Software Metrics: Establishing a Company-Wide Program, Englewood Cliffs, N.J.: Prentice-Hall, 1986.
IFPUG, IFPUG Counting Practices Manual, Release 4.1, Westerville, Ohio: International Function Point Users Group, 1999.
Jones, C., Programming Productivity, New York: McGraw-Hill, 1986.
Jones, C., "Critical Problems in Software Measurement," Burlington, Mass.: Software Productivity Research, 1992.
Jones, C., Assessment and Control of Software Risks, Englewood Cliffs, N. J.: Yourdon Press, 1994.
Jones, C., Applied Software Measurement, Assuring Productivity and Quality, 2nd ed., New York: McGraw-Hill, 1997.
Jones, C., Software Assessments, Benchmarks, and Best Practices, Boston: Addison-Wesley, 2000.
Kemerer, C. F., "Reliability of Function Point Measurement: A Field Experiment," Massa-chusetts Institute of Technology Sloan School of Management Working Paper 3193-90-MSA, January 1991.
Kemerer, C. F., and B. S. Porter, "Improving the Reliability of Function Point Measurement: An Empirical Study," IEEE Transactions on Software Engineering, Vol. 18, No. 11, Novem-ber 1992, pp. 1011–1024.
Littlewood, B., and L. Strigini, "The Risks of Software," Scientific American, November 1992, pp. 62–75.
MacLeod, J. M., "Implementing and Sustaining a Software Inspection Program in an R&D Environment," Hewlett-Packard Journal, June 1993, pp. 60–63.
Myers, G. J., The Art of Software Testing, New York: John Wiley & Sons, 1979.
Oman, P., and S. L. Pfleeger (ed.), Applying Software Metrics, Los Alamitos: IEEE Computer Society Press, 1997.
Shooman, M. L., Software Engineering: Design, Reliability, and Management, New York: McGraw-Hill, 1983.
Sprouls, J., IFPUG Function Point Counting Practices Manual, Release 3.0, Westerville, Ohio: International Function Point Users Group, 1990.
Symons, C. R., Software Sizing and Estimating: Mk II FPA (Function Point Analysis), Chichester, United Kingdom: John Wiley & Sons, 1991.