Economics of Software Quality: An Interview with Capers Jones, Part 2 of 2 (Podcast Transcript)
This is a transcript of an audio podcast.
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Welcome to OnSoftware, conversations with the industry's leading experts on a wide range of programming and development topics. You can access related resources, recommended reading, special offers, and more when you visit the OnSoftware resource center at InformIT.com/softwarecenter.
Rex Black and Capers Jones explore issues of software quality and how to measure it, as discussed in Jones' new book with Olivier Bonsignour, The Economics of Software Quality. In part 1 of this series, they covered the importance of learning how to measure quality in software development and why metrics for quality should be applied early in the process—long before testing begins.
Rex Black: Now, how do you think that we as a profession are doing, in terms of capturing information about defects? Not just steps to reproduce them, that allow the developers to then fix those defects, but also good classification information that allows people to really listen to the software process and its quality capabilities (or lack thereof) and make the kind of intelligent process-improvement decisions that need to be made?
Capers Jones: Well, as an overall industry, we're not doing very well. But there are some major exceptions to that—there are companies that are doing very well. For example, companies like IBM, Motorola, and Boeing—the companies that build large and expensive physical products, like airplanes, computers, and telephone switching systems absolutely need state-of-the-art quality levels, because otherwise these expensive devices won't work and can't be sold. So the engineered products community is actually pretty sophisticated in both measuring quality and achieving it consistently; defect-removal efficiencies in the high 90 percent, as opposed to the 85 percent.
Where you run into major problems worldwide are companies that did not have much in the way of quality assurance, and suddenly found themselves building software without adequate understanding of the need for quality. So I'm talking about insurance companies, banks, general manufacturing companies. The high-tech engineered products companies, like aircraft manufacturers and computer folks—they had formal quality assurance even before there were computers, because the physical devices needed high quality. But banks and insurance companies didn't have any quality assurance, because what they did was basically capable of being controlled by accountants and people who had the knowledge in their heads, and when software was suddenly added to the mix, they didn't realize how important quality was. So they're playing catch-up. What they need is to understand and model their quality control approaches on the best practices of the more-sophisticated companies, like IBM and the defense contractors and the like.
RB: Yeah, good point. One of the things that I've mentioned on a number of occasions is that, from a testing point of view, the state of the typical practice lags the state of the art by a good 30 to 40 years. What I mean by that is if you look at what the top companies—which are the same kind of companies that you were just talking about—are doing in terms of their testing practices (using certified testers, using formal techniques to develop their test cases, and so forth), differentiate that against the vast majority of companies that really kind of stuck back in the '70s—early '70s, sort of pre-Glenford Myers book The Art of Software Testing—and I think that's a fair characterization. Would you say that that doesn't apply just to testing, but is actually true of software quality practices in general?
CJ: Well, I think that it's a general truth that applies to many human activities, and it certainly applies to software. My own data confirms your data, that the leaders are many years ahead of the laggards in terms of understanding and utilizing state-of-the-art methods. But if you look around you at other kinds of things that people do, you discover similar lags in other domains. For example, there was a big gap in adopting paddle wheels and screw propellers for naval boats. There was a big gap in adopting rifles in place of muzzle-loading muskets for armies—it took almost 50 years for rifles to replace muskets. There was a huge gap in accepting concepts like continental drift. So, in a lot of fields, you end up with these gaps.
In fact, there are several books on the scientific revolution, and there are also some psychological studies. They're interesting enough to mention briefly. The psychological studies say that ideas that contradict ideas that you have in your head will initially be disbelieved because your mind can't accept the dissonance, the discontinuity between what you believe and alternate facts. And so you tend to disbelieve evidence until it becomes overwhelming, and then you make a sudden switch to the new paradigm. Right now, when you try to tell someone that you need inspections and pre-test defect removal like static analysis, they probably will resist it, because in their head, they have "Testing is the only thing we need to do," and so will challenge it. What needs to happen is that the evidence needs to become overwhelming, and then you can see a fairly abrupt transition to a new model, in which inspections, static analysis, formal testing, and certified test personnel become the norm, instead of being resisted.
RB: Yes. We've certainly seen this in testing quite a bit—that cognitive dissonance, and the confirmation bias that people have, where they prefer to hear information that confirms what their existing beliefs are, rather than uncritically accepting information that does conflict with what they expect. Exactly as you say, that disbelief, that misapprehension about what's going on generally persists long after sufficient data has built up on projects that any objective outsider would look at and say, "Whoa, this project is in a great deal of trouble." I also think that part of that is that quality professionals—and I'll include test professionals here—don't necessarily do the world's greatest job at presenting and analyzing the information that they do have. Well, basically, turning raw data into information that non-quality and non-test people can understand. Do you have some thoughts on whether that's a fair observation, and, if so, what people can do to try to better convey to their fellow software professionals what the meaning of their findings are?
CJ: Well, I think it's a fair observation, but I don't have an easy solution to solve that problem. The companies that I work with, where the data is professionally analyzed and turned into useful information, often employ statisticians in their quality assurance and project offices, so they have professionals who are conversant with data collection and data conversion. That's not training that's provided to ordinary software engineers, and it really isn't provided to ordinary software managers. So, if you work in a small company, with just the normal complement of software engineers, a few testers, some quality assurance people, and a management cadre, you won't have the internal skills, probably, to do that. But if you work in a big company—an AT&T, a Boeing, a Microsoft, a Google—they have internal statisticians and people who know how to use that data. Unfortunately, smaller companies don't usually afford these specialists to present the data, so they end up not being able to gain access to it.
RB: Yeah. It seems also from my perspective that some of the project management tools that we have, including test management tools, aren't really up to the job, either. They don't provide as much assistance as one could really hope. Often what we see is that people suffer from what I call the "fire hose of data" problem, which is that the test manager or quality assurance manager goes into project status meetings and just unleashes this torrent of raw data—test case counts, test case status, bug counts, bug arrival rates, and so forth—up to 20 or so graphs and charts of information that is far too tactical for the other people sitting in the room to understand, because it's really all just about test project status. None of it really relates back to what was to be tested and what the conclusions are in terms of quality. Do you have any thoughts on where we are in terms of tools to support the kind of economic analysis of software quality information that you talk about in your book? Do you see any promising trends there?
CJ: Well, I've seen the same phenomenon you do—that test managers and others present tons of data, but it doesn't actually show any conclusions, or make top executives aware of what the real problems are. In fact, that's one of the reasons why I wrote this book with Olivier on the economics of software quality; it's to try to clarify where the real value is from things like inspections, pre-test static analysis, certified testers, better test case design—and make that data easily available to people who may not have internal statisticians of their own. By publishing it, I thought I would kind of provide the data to companies that aren't able to collect it themselves. Now, in terms of the kinds of tools that are needed to do that, there is a relatively sophisticated set of companies that build predictive estimating models that can predict quality as well as cost and schedules. Companies like Galorath (SEER); PRICE Systems (TruePlanning), my old company Software Productivity Research (KnowledgePLAN)—all of these tools are relatively sophisticated in predicting quality and the cost of quality. The companies that use them can get a pretty good advance warning about what they need to do because they're sensitive to the utilization of things like inspections and static analysis. But the problem there is that that entire sub-industry of companies that build these tools—the whole set of companies together—probably have only reached 25 to 30 percent of the potential client base, so there's still 65, 70, 75 percent of companies that don't use these tools.
RB: Right, right. Yeah. Just understanding the importance of being able to predict in advance the number of defects that you're gonna have to deal with on a project is certainly more exceptional than one would want it to be.
CJ: Well, it is exceptional, but there's a very easy rule of thumb that will give you a good first-order approximation. If you take the function-point total of an application, and raise that to the 1.25 power, that will give you an alarmingly good prediction of the probable number of bugs that are gonna be encountered over the life of the project.
RB: I agree. It's not that it's not possible to do reasonably good estimations, and I've seen that particular rule of thumb work, as you say, quite well. And I have seen clients that have put together spreadsheets, based on historical data, that do a reasonable job of defect prediction. But it's surprising that, given how relatively simple that [sort of prediction] is, how few people do it. That's one of the things that I really hope about the publication of your book—that it will raise awareness that, hey, this is a solvable problem. This is not something we have to live with. Software quality is not something we're doomed to suffer from problems in that area. Is that something that you can see as an outcome of your book being published?
CJ: Well, I did publish the book because I wanted to point out that there were technologies available today, that, if they're used, can give you a very synergistic combination of higher quality, lower costs, and happier customers—and do that all at the same time. But, unfortunately, it's almost in a class like a self-help book, like a dieting book or an exercise book—it will help the people that read it and take it seriously, but millions of people may not read it, and therefore may not know about what the solutions are.
RB: Yeah. In addition to some of the things that we've discussed, one of the things that jumped out at me in reading your book is all the remarkable figures in there—shocking figures, often, about the economics of software quality. We've discussed a few of those. In addition to the ones that we've discussed, what would you say are some of the other really big "money pits" in terms of software quality? Do you have some thoughts about how software professionals can go about reducing losses [they] suffer?
CJ: Well, there are many kinds of quality, and they have different solutions to them. The form of quality that I tend to deal with is traditional functional quality, where you're looking at bugs and defects and the most effective ways of eliminating them. But there's also structural quality, which my coauthor Olivier dealt with, which deals with whether your application works well in a multi-tiered environment, how well it communicates and cooperates with the operating system, with related applications, and with the databases that are providing it the data. And then, finally, there is aesthetic quality. That's the cosmetic appearance of the screens and the reports, and the information that goes in and out of it. That [quality] is normally measured using opinion surveys, because aesthetics are kind of outside the area that can be carefully quantified, so you can only deal with opinion surveys. But you need to put all of those things together to form a cohesive whole if you want to optimize functional quality, structural quality, and aesthetic quality at the same time.
Which brings up the point: Some of the same large companies that have statisticians also have graphics artists and cognitive psychologists who help in designing optimal interfaces, like Microsoft and Google and IBM have psychologists as well as statisticians available inside. Bottom line is that big and sophisticated companies have done a good job, because they have people—specialists—who can help them do a good job. Smaller companies, or companies in industries like insurance and banking, that lack these internal specialists, are still groping for solutions.
RB: This sort of brings us back to the historical points that you were making earlier. It sounds like, thinking back to, say, the history of the early Industrial Revolution, when small groups of people—sometimes one or two people—got together and had great ideas, and were able to put together machines that, for their time, were remarkable. But if you think back to, say, the first steam engines now, they are obviously very crudely engineered products. So it seems like the implication here is that we in the software engineering field are still in that early phase where people sort of bang things together, and they kind of work, but they don't have a lot of quality—whether we're talking about functional, structural, or aesthetic quality. Do you see us on a similar trajectory, as engineers, and can we expect that the next 50 years will bring us maturation of software engineering?
CJ: Well, I think we will mature over 50 years. But if you look at the history of computing and software from the 1960s forward, when I got into the industry in the late '60s, there were a lot of computer manufacturers—IBM, RCA, Honeywell, Bull, and quite a few others—Control Data Corporation, Digital Equipment. One of the more interesting large-scale industrial studies demonstrated that one of the reasons that IBM tended to dominate the others was because IBM had better quality, and better customer service and customer support than their major competitors. So they were able to ship complicated products with fewer bugs, and then fix them faster (when customers found those bugs and had trouble with them) than their competitors. And not only for computers, but for other kinds of manufactured products, such as automobiles or television sets, that same phenomenon tends to lead to increases in market share. In other words, the companies with the best quality, the lowest delivered defect rates, and the best customer support end up surviving and doing well, whereas companies with similar products that don't have good customer support don't do so well. In fact, a current proof of the existence of this phenomenon is Apple. Every time you look at something that evaluates customer support, Apple turns out to be near the top of the list—and look at their market capitalization and value. What's happened to Apple primarily is a result of building innovative products, but also making sure that they work pretty well and have good support when they're released.
RB: Good point. Even having been in this field, testing and quality, for years myself, your book has certainly helped raise my awareness of the magnitude of the quality situation with software. I hope a lot of people read it, so that a lot more people do become aware. I know you mentioned that it's kind of like a self-help book—it can only help those who pick it up—and I hope a lot of people do. Have you seen signs over the span of your career, Capers, that we are as an industry gradually becoming more aware of the financial side of software quality?
CJ: Well, yes, I think in general we are becoming more aware of it. That doesn't mean that that trend is moving as fast as I would like to see it, but it certainly seems to be moving. And what may be a little bit alarming is that it tends to be moving faster in some [other] countries than it does here in the United States. For example, if you look overseas, India now has more companies certified to CMMI level 5 (that's the Capability Maturity Model developed by the Software Engineering Institute) than any other country—I believe including the U.S. So other countries are alerted to the fact that quality is a critical factor for increased market share, and tend to be moving at least as fast as, and possibly faster than us.
RB: Yeah, we've seen that trend with certified testers, too. There are about 250 percent more certified testers in India than there are in the United States.
CJ: Well, we also saw, for example, the rapid increase in the Japanese economy, not long after World War II; when, as a result of Deming and Juran and other people, they began to take quality seriously in their manufactured products, and suddenly catapulted ahead of the United States, with quality of automobiles, television sets, consumer electronics—and therefore also catapulted ahead of the United States in terms of sales of those products.
RB: Yeah, it seems like human beings in general are not necessarily very good at extrapolating from lessons learned by other groups, in this case other industries, into their own group. Certainly if software professionals would look at that lesson, as offered by the situation you've just mentioned, the Japanese predominance in manufacture in the '80s, a lot of the mistakes that we're making in terms of software quality would be clearer, and presumably would be resolved.
We've covered a lot of ground here, but are there any other things that you'd like to bring up to listeners before we close out this podcast about your book and things that they can glean from reading it?
CJ: Well, one small point. One final point. When I worked at ITT, I was part of the Corporate Quality Council; that's the thing that'd been started by Phil Crosby, whose famous book Quality Is Free makes the point that quality is free. And not only for software is quality free, but it actually has a positive return on investment. So by emphasizing quality in software, you get more than just a free ride. You actually gain market share, you lower your warranty costs, you shorten your development schedules, and you improve team morale. You get a relatively powerful return on investment for a relatively small outlay of cash and energy. So I think software can actually go beyond the concept of "quality is free," to the concept that "quality pays a handsome profit."
RB: That's a great summary, and a great point to end on. The sooner more people start thinking that way, the better off the software industry will be, certainly.
Well, thank you very much, Capers, for the opportunity to talk to you today. I've really enjoyed our discussion, and I hope you have as well.
CJ: Well, thank you, Rex. I enjoyed it. Thanks for the questions, and thanks for being a reviewer of the book and for the opportunity to speak with you.
RB: It was my pleasure.