Software Assessments, Benchmarks, and Best Practices

Description

• Copyright 2000
• Dimensions: 7-3/8" x 9-1/4"
• Pages: 688
• Edition: 1st

• Book
• ISBN-10: 0-201-48542-7
• ISBN-13: 978-0-201-48542-4

Traditionally, qualitative software assessments and quantitative benchmarking have been performed separately -- if at all. Moreover, many quantitative benchmarks have been proprietary secrets. Now, world-renowned authority Capers Jones has integrated qualitative and quantitative approaches into the first complete methodology for identifying best -- and worst -- software development practices. Readers will discover more effective ways to leverage assessment techniques such as the Capability Maturity Model (CMM); learn the best benchmarks and metrics; and link quantitative benchmarks to qualitative assessments. The methodology presented in this book gives developers and managers a powerful new way to project and evaluate the productivity and quality improvements associated with changes in software processes. The result: better quality, achieved more quickly and at lower cost.



Sample Content

Table of Contents

Preface.


Acknowledgments.


1. Introduction.

Protecting and Comparing Confidential Data.

International Country and City Codes.

Using Standard Industry Classifications (SICs) for Software Studies.

A Taxonomy of Software Projects for Benchmark and Assessment Studies.

2. Software Process Assessments.

The Origins of Software Process Assessments.

The SEI Assessment Approach.

The SPR Assessment Approach.

Pattern of Client Strengths.

Pattern of Average Results.

Correlating the SEI and SPR Scoring Systems.

Readings and References on Software Assessments.

3. Software Benchmarks and Baselines.

Benchmarks and the Problem of Size Metrics.

Benchmarketing Software Costs.

Hazards of Using Project-Level Data for Software Benchmark Studies.

Hazards of Mailed Surveys for Software Benchmark Studies.

Moving to Activity-Based Software Benchmark Data.

Software Quality Benchmarks.

Software Baselines.

Readings and References on Software Benchmarks and Baselines.

4. Thirty-Six Key Factors for Software Assessment and Benchmark Studies.

Software Classification Factors.

Project-Specific Factors.

Technology Factors.

Sociological Factors.

Ergonomic Factors.

International Factors.

Readings and References on Factor Analysis.

5. Identifying Software Best and Worst Practices.

Factors that Influence Software Development Productivity.

Factors that Influence Software Maintenance Productivity.

Patterns of Positive and Negative Factors.

Readings and References on Software Best and Worst Practices.

6. Software Process Improvements.

Annual Tactical and Strategic Software Improvement Plans.

Annual Software Progress Report.

The Six Stages on the Path to Software Excellence.

Stage 0: Software Process Assessment, Baseline, and Benchmarks.

Stage 1: Focus on Management Technologies.

Stage 2: Focus on Software Processes and Methodologies.

Stage 3: Focus on New Tools and Approaches.

Stage 4: Focus on Infrastructure and Specialization.

Stage 5: Focus on Reusability.

Stage 6: Focus on Industry Leadership.

The Costs, Timing, and Value of Process Improvements.

Readings and References on Software Process Improvements.

7. Benchmarks And Best Practices For Mis Software Projects.

MIS Applications and Corporate Databases.

Lack of Data Metrics.

MIS and Enterprise Resource Planning.

MIS Applications, the Euro, and the Year 2000.

MIS Technologies.

Diverse Origins of Information Systems and Systems Software.

MIS Demographics.

MIS Benchmarks.

MIS Software Successes and Failures.

MIS Success Factors.

MIS Failure Factors.

Best Technical Practices for MIS Software.

Best Project Management Practices for MIS Software.

Best Requirements-Gathering and Analysis Practices for MIS Software.

Best Design and Specification Practices for MIS Software.

Best Coding Practices for MIS Software.

Best Reusability Practices for MIS Software.

Best Change Control Practices for MIS Software.

Best User Documentation Practices for MIS Software.

Best Quality Control and Pretest Defect Removal Practices for MIS Software.

Best Testing Practices and Tools for MIS Software.

Best Maintenance and Enhancement Practices for MIS Software.

Best Personnel Practices for MIS Software.

Best Staff Hiring Practices for MIS Software.

Best Staff Training and Education Practices for MIS Software.

Best Management Training and Education Practices for MIS Software.

Best Practices for Specialization in the MIS Software Domain.

Best Compensation and Salary Practices for MIS Software.

Best Practices in Office Ergonomics for MIS Software.

Best Practices in Organizational Structures for MIS Software.

Best Practices for Employee Morale in MIS Software.

Best Work Patterns and Overtime Utilization Practices for MIS Software.

Best Practices for Minimizing MIS Staff Turnover Rates.

Summary and Conclusions on MIS Software.

Readings and References for MIS Software.

8. Benchmarks and Best Practices for Outsourced Software.

The Growth of International Outsourcing.

Choosing Outsourcing or In-house Development.

Minimizing the Risks of Disputes Between Outsource Contractors and Their Clients.

Outsourcing Technologies.

Outsource Demographics.

Outsource Benchmarks.

Outsourcing Successes and Failures.

Outsourcing and Contract Software Success Factors.

Outsourcing and Contract Software Failure Factors.

Best Technical Practices for Outsourced Software.

Best Project Management Practices for Outsourced Software.

Best Requirements-Gathering and Analysis Practices for Outsourced Software.

Best Design and Specification Practices for Outsourced Software.

Best Coding Practices for Outsourced Software.

Best Reusability Practices for Outsourced Software.

Best Change Control Practices for Outsourced Software.

Best User Documentation Practices for Outsourced Software.

Best Quality Control and Pretest Defect Removal Practices for Outsourced Software.

Best Testing Practices and Tools for Outsourced Software.

Best Personnel Practices for Outsourced Software.

Best Staff Hiring Practices for Outsourced Software.

Best Staff Training and Education Practices for Outsourced Software.

Best Management Training and Education Practices for Outsourced Software.

Best Practices for Specialization in the Outsource Software Domain.

Best Compensation and Salary Practices for Outsourced Software.

Best Practices in Office Ergonomics for Outsourced Software.

Best Practices in Organizational Structures for Outsourced Software.

Best Practices for Employee Morale Among Outsource Vendors.

Best Work Patterns and Overtime Utilization for Outsourced Software.

Best Practices for Minimizing Outsourcing Staff Turnover Rates.

Summary and Conclusions on Outsourced Software.

Readings and References for Outsourced Software

9. Benchmarks and Best Practices for Systems Software.

Systems Software Demographics.

Systems Software Benchmarks.

Systems Software Successes and Failures.

Systems Software Success Factors.

Systems Software Failure Factors.

Best Technical Practices for Systems Software.

Best Project Management Practices for Systems Software.

Best Requirements-Gathering and Analysis Practices for Systems Software.

Best Design and Specification Practices for Systems Software.

Best Coding Practices for Systems Software.

Best Reusability Practices for Systems Software.

Best Change Control Practices for Systems Software.

Best User Documentation Practices for Systems Software.

Best Quality Control and Pretest Defect Removal Practices for Systems Software.

Best Testing Practices and Tools for Systems Software.

Best Maintenance and Enhancement Practices for Systems Software.

Best Personnel Practices for Systems Software.

Best Staff Hiring Practices for Systems Software.

Best Staff Training and Education Practices for Systems Software.

Best Management Training and Education Practices for Systems Software.

Best Practices in Office Ergonomics for Systems Software.

Best Practices in Organizational Structures for Systems Software.

Best Practices for Employee Morale in Systems Software.

Best Work Patterns and Overtime Utilization for Systems Software.

Best Practices for Minimizing Staff Turnover Rates.

Summary and Conclusions on Systems Software.

Readings and References for Systems Software.

10. Benchmarks and Best Practices for Commercial Software.

Special Issues and Technologies for Commercial Software.

Commercial Software Demographics.

Commercial Software Benchmarks.

Commercial Software Successes and Failures.

Commercial Software Success Factors.

Commercial Software Failure Factors.

The Microsoft Pattern.

The Emergence of the ISO 9000-9004 Standards.

Best Technical Practices for Commercial Software.

Best Project Management Practices for Commercial Software.

Best Requirements-Gathering and Analysis Practices for Commercial Software.

Best Design and Specification Practices for Commercial Software.

Best Coding Practices for Commercial Software.

Best Reusability Practices for Commercial Software.

Best Change Control Practices for Commercial Software.

Best User Documentation Practices for Commercial Software.

Best Quality Control and Pretest Defect Removal Practices for Commercial Software.

Best Testing Practices and Tools for Commercial Software.

Best Maintenance and Enhancement Practices for Commercial Software.

Best Personnel Practices for Commercial Software.

Best Staff Hiring Practices for Commercial Software.

Best Staff Training and Education Practices for Commercial Software.

Best Management Training and Education Practices for Commercial Software.

Best Practices for Specialization in the Commercial Software Domain.

Best Compensation and Salary Practices for Commercial Software.

Best Practices in Office Ergonomics for Commercial Software.

Best Practices in Organizational Structures for Commercial Software.

Best Practices in Employee Morale in Commercial Software.

Best Work Patterns and Overtime Utilization for Commercial Software.

Best Practices for Minimizing Staff Turnover Rates in the Commercial Software Domain.

Summary and Conclusions Commercial Software.

Readings and References for Commercial Software.

11. Benchmarks and Best Practices for Military Software.

Military Software Technologies and Special Problems.

The Software Engineering Institute.

The Airlie Council.

Military Software Demographics.

Military Software Benchmarks.

Military Software Successes and Failures.

Military Software Success Factors.

Military Software Failure Factors.

Best Technical Practices for Military Software.

Best Project Management Practices for Military Software.

Best Requirements-Gathering and Analysis Practices for Military Software.

Best Design and Specification Practices for Military Software.

Best Coding Practices for Military Software.

Best Reusability Practices for Military Software.

Best Change Control Practices for Military Software.

Best User Documentation Practices for Military Software.

Best Quality Control and Pretest Defect Removal Practices for Military Software.

Best Testing Practices and Tools for Military Software.

Best Maintenance and Enhancement Practices for Military Software.

Best Personnel Practices for Military Software.

Best Staff Hiring Practices for Military Software.

Best Staff Training and Education Practices for Military Software.

Best Management Training and Education Practices for Military Software.

Best Practices for Specialization in the Military Software Domain.

Best Compensation and Salary Practices for Military Software.

Best Practices in Office Ergonomics for Military Software.

Best Practices in Organizational Structures for Military Software.

Best Practices in Employee Morale in Military Software.

Best Work Patterns and Overtime Utilization for Military Software.

Best Practices for Minimizing Military Staff Turnover Rates.

Summary and Conclusions on Military Software.

Readings and References for Military Software.

12. Benchmarks and Best Practices for End User Software.

End User Software Demographics.

End User Benchmarks.

End User Software Successes and Failures.

End User Software Success Factors.

End User Software Failure Factors.

Best Technical Practices for End User Software.

Best Personnel Practices for End User Software.

Readings and References for End User Software.

Appendix: SPR Questionnaire for Assessments, Benchmarks, and Baselines, and Appendix Glossary.


Complete List of Readings and References.


Index. 0201485427T04062001

Preface

During my writing, this book evolved considerably from the first plan. Originally I intended to divide the book into two major sections. The first section was to discuss a number of assessment and benchmark methods used in the United States and Europe. The second section was to present an overview of software productivity and quality benchmarks, and associated "best practices" derived from benchmark studies. The benchmarks and best practices in this book cover six major kinds of software project: (1) management information system (MIS) projects, (2) outsource projects, (3) systems and embedded software projects, (4) commercial software projects, (5) military software projects, and (6) personal software projects developed by end users.

However, as the writing commenced, the focus of the book began to change. It soon became clear that a complete discussion of benchmarks and best practices for each of the six kinds of software would be about twice as large as initially planned. I had planned to devote approximately 30 pages to the benchmark and best-practice information for each type of software. But to do justice to the available data, almost 60 pages were needed for five of the six forms of software. Furthermore, a discussion of how assessment and benchmark studies operate and their technical differences may be of interest to those of us in the assessment and benchmark business, but it is not necessarily of great interest to those outside the limited circle of benchmark consultants.

As a result, the discussion of assessment and benchmark methods was cut back, and the sections devoted to information gathered during assessments and benchmark studies was expanded. Instead of a book with two sections of roughly equal size, the book now has a briefer introductory section and greatly expanded discussions of each type of software and the issues that confront each type.

This book also emphasizes assessments and benchmark data from the United States. Although my colleagues and I have gathered data in more than 24 countries, the issues of international benchmarks are quite complex. The international variations in working years and working days, how overtime is treated, and European restrictions on some kinds of data collection made me decide to concentrate on U.S. data.

Readers should note that this is a book about assessments and benchmarks written by someone who is in the assessment and benchmark business. Because my company has been performing assessments and benchmarks since 1985, we have an obvious interest in the topic. However, this is not a marketing book, nor is it a book about how my company's assessments and benchmarks work. The topics of software assessments and benchmarks are important ones, and this book attempts to include the general principles under which all assessment and benchmark consulting groups operate.

In my view, and also in the view of my competitors, assessments and software benchmarks are important to the global economy. Software is the most labor-intensive product of the twentieth century, and the most error prone. Assessments, benchmarks, and empirical data are on the critical path to minimizing software project failures. Every software project manager, every software quality assurance specialist, and every software engineer should understand the basic concepts of software assessments and benchmarks. This is a view shared by all of the assessment and benchmark consulting groups.

The software industry has achieved a notorious reputation as being out of control in terms of schedule accuracy, cost accuracy, and quality control. A majority of large systems run late, exceed their budgets, and many are cancelled without ever reaching completion. Assessment and benchmarks followed by planned process improvement programs can aid in bringing software under management control. These are not "silver bullet" methods. Assessments, benchmarks, and process improvement programs require effort and can be expensive, but project failures are far more expensive.

This book discusses the kinds of complex software projects that benefit from assessments and benchmark studies. Small and simple projects are not the main focus of assessments and benchmarks. The proper focus of assessments, benchmarks, process improvements, and this book is on large and complex applications.

Chapter 1 provides an introduction to the topic of software assessments and benchmarks. This chapter discusses the kinds of data that should be collected. It also cautions against some common problems, such as depending on data without validating it, and using hazardous metrics such as lines of code. This chapter also discusses the need to keep client data protected, and suggests some coding methods that can be used to perform benchmarks without revealing proprietary client information.

Chapter 2 deals with the history of software process assessments and discusses some of the kinds of information that is gathered during software process assessments. Although more than a dozen forms of assessment exist, the form made popular by the Software Engineering Institute is the best known. Some recent and more specialized forms of assessment, such as those performed for the year 2000 problem, have also been widely used since about 1998.

Chapter 3 deals with the related topics of software benchmarks and software baselines. Benchmarks collect and compare quantitative data against industry norms. Baselines measure the rate at which a company can improve productivity and quality when compared with an initial starting point. Of course, sometimes productivity and quality can get worse instead of better.

Chapter 4 discusses 36 key factors that should be recorded during assessment and benchmark studies. If these 36 key factors are recorded, the data gathered by almost any benchmark and assessment consulting group, or by any company or government agency, could be compared meaningfully.

Chapter 5 addresses an important topic that is somewhat ambiguous in the software literature. When we speak of "best practices" what exactly do we mean? Chapter 5 discusses some criteria for including or excluding tools and technologies from best-practice status. It is suggested that any technology considered a potential best practice needs empirical results from at least ten companies and 50 projects.

Chapter 6 discusses an important follow-on activity to assessment and benchmark studies. Both assessments and benchmark studies are diagnostic in nature, rather than therapeutic. These studies can identify problems, but they cannot cure them. Therefore, a natural follow-on activity to either an assessment or a benchmark analysis, or both, would be to implement a process improvement program.

Chapter 7 deals with benchmarks and best practices for MIS projects. These are software applications that companies and government agencies build for their own internal use. MIS applications are often keyed to large corporate database access, and their main purpose is to convert raw data into useful information. Although MIS projects are often fairly high in productivity for small projects, large MIS projects tend to experience more than average failure rates. Quality at the large end is often poor too.

Chapter 8 deals with benchmarks and best practices for outsource software projects. The emphasis in this chapter is on projects under contract for MIS, rather than for military or systems software outsourcing. The major outsource vendors such as Andersen, Electronic Data Systems, and IBM concentrate on the MIS market because it is the largest market for their services. In general, outsource projects have higher productivity and quality levels than in-house MIS projects; however, litigation between clients and outsourcers does occur from time to time.

Chapter 9 deals with benchmarks and best practices for systems and embedded software, which are applications that control physical devices such as computers, telephone switching systems, aircraft flight controls, or automobile fuel injection systems. The close coupling of systems software to physical hardware devices has led to very sophisticated quality control methods. The systems software community has the best track record for large applications larger than 10,000 function points.

Chapter 10 deals with benchmarks and best practices for commercial software. Commercial software applications are intended for the mass market, and some of these applications are used by millions of customers on a global basis. The commercial and systems software domains overlap in the arena of operating systems because commercial products such as Windows 98 are both systems and commercial software. The commercial world needs to deal with special issues such as translation and nationalization of packages, piracy, and very extensive safeguards against viruses.

Chapter 11 deals with benchmarks and best practices for military software, with special emphasis on the U.S. armed services and the Department of Defense. The military software domain is fairly good at building large and complex applications, although military software productivity is lower than any other domain. The legacy of U.S. military standards has left the defense community with some very cumbersome practices. Plans and specifications in the military domain are approximately three times larger than equivalent civilian projects. The bulk is due primarily to military oversight requirements, rather than to the technical needs of the project.

Chapter 12 deals with benchmarks and best practices for end user software development. As the century ends, there are more than 12,000,000 U.S. office workers who know how to write computer programs if they wish to do so. By the middle of this century, the number of computer-literate workers in the United States will top 125,000,000. Indeed, there are some signs that computer literacy will actually pull ahead of conventional literacy in the sense of being able to read and write. End user applications are currently in a gray area outside the scope of normal assessments and benchmarks. More importantly, end user applications are also in a gray area in terms of intellectual property law. As end user applications become more and more numerous, it is important to set policies and guidelines for these ambiguous applications.

As this book is written, benchmarks based on function point metrics are dominant in the software world, except for military software, in which benchmarks based on lines of code still prevail. This book utilizes function point metrics and cautions against lines-of-code metrics for benchmarks involving multiple programming languages. Version 4.1 of the function point rules defined by the International Function Point Users Group is the standard metric used throughout.

0201485427P04062001



Updates

Submit Errata

