An Interview with Watts Humphrey, Part 27: Formal Methods, the Failure of PSP, and Tragedy of Beautiful Software

Like this article? We recommend

Reflections on Management: How to Manage Your Software Projects, Your Teams, Your Boss, and Yourself

Learn More Buy

Like this article? We recommend

Like this article? We recommend 

Reflections on Management: How to Manage Your Software Projects, Your Teams, Your Boss, and Yourself

Learn More Buy

Formal Methods

Booch: A question for you with regards to the defect-free programming, because my recollection is that this was around the same time that Dr. Tony Hoare was talking about the notions of formalisms in development. Edsger W. Dijkstra of course as well around that time. And their thought was, you know, really formal specifications of the programs, but you were looking at an alternate view with regards to process. How did that fit in? What knowledge did you have of what they were up to?

Humphrey: I got fascinated with the formalisms and it was very attractive. I took a tutorial with Tony at ICSE, I had a group working in Germany while I was in my IBM quality process job. I used to go and visit with the people, particularly in Germany, because people there, they were doing marvelous work. A man named Horst Remus at IBM -- retired about the time I did -- introduced me to a bunch of these people, and they were academics they were working with in Germany, and they had these formal design methods. I also ran into some in the Netherlands that were doing that. So I was looking at that, the Zed language, if you’re familiar with that.

Booch: Very much, yes.

Humphrey: So I got interested in that, and I actually tried some of these methods on my own work, and I found it was quite helpful -- the discipline was helpful -- but I found the problem was I wasn’t that familiar with the formalisms, and that I tended to make more errors when I did it that way than I did otherwise. Part of the reason was that I wasn’t that good with the formalism stuff, so familiarity with the language was a real problem for me. But the other was that I then had to translate it into code, and I made a lot of errors there too. So my error rates actually went up rather significantly when I tried to do that, and I think part of it was my own problem. I think the formalism approaches will work at the individual level, and I think that’s useful. I don’t know if you’re familiar with the people with the SPARK programs at Praxis in England. Have you heard about those?

Booch: I have. There’s John. One of the early Ada guys was involved in that one.

Humphrey: Oh yeah. What they did is they took an Ada subset, and they subsetted Ada to make basically what they concluded was a precise language that didn’t have all of the misunderstandings and confusions in a lot of the languages. I actually went and took their course, their SPARK course, in England.

Booch: It was John Barnes. I think that was the guy still involved with it.

Humphrey: Well, I just got an e-mail from one of the guys the other day. They’ve now made their whole language and toolset available open source, which is marvelous. And they’re having good experience. One of their top engineers-- Rod Chapman I think is his name -- I went over there and he taught me and Noopur Davis, one of our people, and we had some other people in the class. He actually taught us the SPARK method, and we wrote some programs and went through and did it. It was an eye-opener. To identify all sources of buffer overflow is a hell of a challenge, and the mathematical methods are extremely helpful in doing that. So I found that I’m very impressed with those methods; I think they’re very powerful. And Rod Chapman actually came over and got trained as a PSP instructor, a TSP coach. So he’s qualified in the PSP and TSP. And Rod and I have discussed this at some length, and we both believe that the two methods are quite complementary.

They both address important but slightly different facets of the problem. And we both believe that the two methods really ought to get merged. One of the problems, of course, with SPARK is that it’s an Ada subset language, so that’s a problem. Later on, when I talk about the future, I want to talk about languages. So I agree, the formalism stuff is great. But what was amazing to me was we can get such an enormous level of improvement without it. And that’s what I’ve found is amazing, is we go through training engineers with the PSP, just gathering data on their defects, just looking at it, analyzing defects.

The reason it’s so extraordinary is that when you make a mistake-- and I tried to eliminate my mistakes, and I literally couldn’t do it. I found that there was no way that I could write programs without making errors. And the reason was that I was human, contrary to popular opinion, right? But there was no way I could literally do it. Because I was human, I tended to make various kinds of errors. But the thing that was interesting about it was I tended to make a particular category of errors. So I’d make them repetitively, and what surprised me was that if I gathered data on the kinds of mistakes I had made and used that data to put together a checklist to go back and review my code to find those errors, I got damn good at it. I got to where I could find most of the errors before I even compiled my program. I didn’t get to where I could find all of them. I did occasionally. But I find really good programmers do get that good.

The Yield Measure

What we’ve got in the PSP is the measure called “yield,” and you have the yield data. What percentage of the defects do you find in compile, and what percent do you find in the testing, what percent do you find in inspection or a code review, and that sort of thing. What I’ve found as I’ve looked at the data, on code review yields, with a lot of work, you can get up to 70 to 80 percent. I was typically getting close to 80 percent yields on my code reviews, which is pretty good. I find most people don’t even bother doing a code review, so most of the defects just slip through. You find also compiler yields are typically in the 50 percent range or less. I find a surprising number where the defects that you would classify as syntax defects, but they’re not really true syntax defects. It’s like a misspelled word that turns out to be a valid other word. So you make syntax defects that turn out to produce valid syntax. So it’s not really a syntax defect, but it’s that kind of a problem. It’s a typo; it’s a whatever. But it ends up producing valid code. It will compile.

So one of the things we try to convince people in the PSP course is that compilers or builders, whatever you got -- I know many people are getting away from compilers -- but whatever that tool is, its job is to try to take whatever you produce and make running code out of it. So it’s not really trying to find errors; it’s trying to produce code. So to count on any automated code-building tool like that to find errors for you is a mistake. Now, that isn’t true of things like the SPARK tools, which are designed to find errors, that really analyze it, and they’re looking at it. But that certainly is true of most of the code-building tools.

So that is really kind of a key question. You really have to focus on the quality of what the individual does. What I’ve found as part of that, that the various kinds of quality reviews, I do personal reviews. I review designs; I review the code. So I produce a program, for instance, and suppose I spend an hour writing the code. I force myself to spend at least 30 minutes going over the code, going over it carefully to try to find every error. And as I said, I can find 70 to 80 percent of them, or I could when I was writing programs a lot. So people can get pretty good at it. But what’s interesting is the defects I find personally when I do reviews, I’ll find them quickly, but I’ll find quickly a lot of the defects that would take you forever to find in tests.

Conversely, defects that are bloody hell to find individually pop up right away in tests. Like the blue screen -- all that stuff. It doesn’t work at all. Well, some of those things you completely miss -- the systemic-level stuff shows up right away in test, but you can’t see it. So there are a whole lot of issues like that that I find are kind of fascinating. And the thing that’s amazing to me is if you take and improve the quality of the code going into test by 10 times, you’d think you’d improve the quality coming out of test by 10 times. What’s amazing is you don’t. You improve it by close to 100 times. Sort of amazing.

So it takes a relatively modest investment, but we’re getting extraordinary improvements in quality by really addressing the individuals and getting them conscious of the quality of their personal work, and giving them data on it so they can look at it. So that’s what we did with the PSP, and that’s what I’ve found. So I taught the PSP course at CMU, and I think I mentioned that four of the people who took the course got so excited about it that they decided to change their lifetime careers, and I’ve got people working with me now that were in that course.

So it’s a transforming experience to go through this, and we have thousands of engineers who have now done it. And with very few exceptions, they come out of this saying, “Wow. That’s the way I want to work.” I’d say about, oh, 60-70 percent come about with a sort of “wow” feeling. Another 30-40 percent come out with an, “Well, okay, but I’m not sure I can do it on the job” feeling. And then there’s a small number of hardheads that really don’t buy it. But it’s kind of interesting. But in any event, so we got the PSP going, and we started to teach classes and bring people in to take courses.

I worked with Embry-Riddle University, and they gave me my very first experience with using the TSP. I put together the process and I talked to the people at Embry-Riddle who -- let me back up before I do that, because I ran into Embry-Riddle people at an SEI symposium. SEI would run a symposium every year. We’d bring in DoD people and industry people, et cetera, and we’d present our plans and what we were doing.

I was in one of these meetings; I made a presentation and I was sitting watching somebody else, and a guy came up and tapped me on the shoulder and said he wanted to talk to me. So I stepped out to see what he wanted. His name was Soheil Kajenuri. He was a professor at Embry-Riddle University, and he said, “We’re building a master’s curriculum at Embry-Riddle, and we’d like to build it around the CMM.” So he said, “I’d like you to talk to us.” I said, “Well, I’ll come out and talk to you. I can’t do it now, but I can come out.” We picked the time.

So he was there with their department chairman, Iraj Hirmanpour, and so I met with them. So I went through what they wanted to do, and they wanted to build a curriculum about the CMM. I said, “Well, basically, doing that’s fine, but you need to have the PSP in there.” So I described all of that, and they got all excited about it and said, “Yeah, that’s a good idea.” So they made the PSP the first required course in the graduate program. And Soheil taught a PSP course shortly thereafter, out of my textbook. I think the book was published by then, I believe, I don’t think it was the manuscript. Although it may still have been a manuscript. This was quite early.

So it was like 1995, ’94, something like that. Some of the first people writing programs in ’93-- that’s when Howie Dow did his. I taught in early ’94. That’s right. And then my discipline with PSP book, which I taught from in January through June of ’94, I was getting the class data, and then using that data to re-do the book. So I rewrote that book that spring and summer and got the manuscript to my publisher that fall, and it was published actually in December of ’94 with a ’95 copyright. By the way, a side point, I worked with Addison-Wesley, and Peter Gordon is my editor on the Managing the Software Process book, the very beginning. He’s been my editor now on nine books. Early on, I used the PSP to write my books. And he told me, he said he’s never had an author who makes every schedule. And I have. I mean, I’ll go through it, I’ll make a plan, and I’ll meet it. Things don’t come out to be completely predictable. I end up completely rewriting books and that sort of thing. My Winning with Software book this just didn’t work on, so I ended up throwing it away and rewriting it three times. So I was predictable on each version. But in any event, so I had the book, Soheil taught from it, and as I say, it was working great until the people at Embry-Riddle started trying to teach local industry people. They got some companies that were interested and they were teaching there, and they had several courses at like Boeing, Motorola, and other places. And I kept going to check, and I discovered that even though they had the course, and the students came out all excited about it, and they’d start using it on the job, it died out very quickly.

So PSP use did not persist. There are various reasons, but fundamentally, the reason was that the engineers said -- I think I mentioned this before -- that they were working alone. I mean, they were trying to do disciplined work. The analogy I use is trying to practice to be a concert pianist in a deaf world. No one will pay attention. Well, Beethoven was deaf when he wrote his Ninth Symphony, but there aren’t many Beethovens in the world. It’s extraordinary that people really, even though they’re convinced, they have a great deal of difficulty working in the environment where their managers are pushing them to do other stuff, everybody’s got to get into test, quality is not a priority, their peers are racing into test, everybody’s getting credit for getting into test early but they are not recognized for doing quality work.

As I said, I talked about this engineer at Siemens, in Siemens research. He did beautiful work, but no one knew. He was doing it all by himself. He was his own audience. No one else was paying any attention. So beautiful work in software does not get noticed. It’s a tragedy. There was one guy I ran into in IBM – Kingston -- exactly the same thing. I remember it was a long story. Remember, one of the things we’d put in place at IBM before I retired there from the quality job, we discovered that over 20 percent of IBM’s APAR (Applied Programming Assistance Request) repairs -- these were customer-reported defect repairs -- over 20 percent of them actually injected defects.

So we would put out fixes that were themselves defective, and the fixing process was in the maintenance organization; it wasn’t done by the developers. So this became a big issue because when we really looked at the data we discovered it was enormously expensive -- like 40% of our programming service costs. I mean, fixes get slapped in. People don’t install them slowly. So when you have fix problems, you’ve got crises. And we had lots of them. So we started something called APAR certification, where before the maintenance people could ship a fix, they had to verify it with the developer. So we put that in place, and we had this procedure everybody was supposed to follow.

I remember, I was up in Kingston, and I was doing a random check, just talking to experienced engineers who were doing development. I called in one engineer they sent in to get a random check to just see. And I used to like to do this, go talk to working engineers, see what they’re doing, what is going on. So I asked him about APAR certification, what he thought about it and how it worked. And he said, “What’s that?” I was really irate. I said, “What do you mean? We’ve got this enormous effort getting APAR certification across the company. Here’s why we’re doing it,” and that sort of thing. And I said, “How come you don’t know about it?” He said, “Well, I’ve got 30 modules in the core of the MVS system that I’m responsible for, and I’ve never had a defect in any of them.” And I said, “You win. You don’t need APAR certification.” Well, no one knew that. We’ve got this guy doing extraordinary work and no one knew it. But that’s what our world is like. You have these people doing extraordinary work and no one recognizes it. It’s terribly sad.

So in any event, I discovered the PSP -- the personal software process -- much as it worked, and much as it enabled people to do truly gorgeous work, very few people really have the personal discipline and the drive and the commitment, and they certainly don’t have the environment to permit them to do that work. So that’s why I concluded that we really had to move from the PSP to the team process. I had to put together a team environment where you could really do beautiful work. And that’s what led us to the TSP. Okay?

Booch: Got it. Very good. Let me level-set here. What year would this have been you’re starting to see the transition to the team software process?

Humphrey: It was 1995 I published the PSP book -- the end of ’94 actually, January ’95. We spent ’95 trying to put all this stuff together, running into all these experiences, and it was in early ’96 it became pretty obvious we had to put together a team process. So I started to construct a TSP, team process.