An Interview with Watts Humphrey, Part 28: Early TSP Trials and the Teradyne Team Launch
This interview was provided courtesy of the Computer History Museum.
Early TSP Trials
Humphrey: As I said, we’d gotten the PSP in place. It became clear that people really weren’t able to use it. So I had to build something that ordinary folk-- not zealots like me -- could actually use on the job. So that, I decided, was a TSP -- a team software process -- and I wanted to put something together. So starting in about January ’96 -- remember I talked about our process development process? I pulled out my process development process, and I started to develop a TSP process
I got it together during the spring and summer, and I was trying to figure out where to use it, and the people at Embry-Riddle said, “Hey, let’s us do it.” So what they wanted to do was to use it for a team of four or five of their graduate students who were going to do a team project. They’d all been PSP trained, now, so they were qualified.
I put together the
process and had it all defined and ready to go, but unfortunately, in
September, when the project was to start, I had to be in
Our processes, by the way, have things called scripts. It’s what I call an operational process. The process isn’t a big document thing that has paragraphs. It has steps in a script which you follow. I’ve got copies or versions of them in my textbooks and stuff. But it’s got a series of steps, and if you read what it says in the steps -- let me pull up a couple here -- they’re very direct. They just tell you what to do, step by step. So it’s not complicated at all. Let me pull up a process: “Schedule planning template: Enter the following.” I mean, it starts off with a purpose. “The purpose is to record the estimate and actual hours expended by a calendar period, and relate the task plan value of the calendar schedule, plan value earned, value stuff, calculated adjusted planned and earned value.” And then, “General: Expand this template or use multiple pages as needed. Complete it in conjunction with the task planning template.” Then I go, “Enter your name, date, week number, when the project started. Enter a week number, typically starting with one. For very small projects, it may be more convenient to use days instead of weeks.” Then the next one.
So basically it goes through -- this is the instructions for the template, and then the actual process script starts with entry criteria, and then planning. “Step one: Produce or obtain a requirement statement. Use the probe method to estimate total new and changed lines of code, required time and the prediction interval. Complete the size estimate template. Use the probe method to estimate the development time required.” So it goes through steps, very straightforward, and we have scripts like that for all the steps. The reason I had to do that is kind of interesting.
And think about it this way: when you tell somebody to produce a plan for a program, most programmers won’t have the vaguest idea what you’re talking about. So they’ve got to go off and sit down and figure out, “How do I make a plan?” They don’t know. They have got to kind of figure it out. Even though it’s not very complicated, they don’t know; they don’t have a process; they don’t know the steps. So they’ve got to figure out step by step what to do, and then they do it. So that’s an enormous waste of time. It’s fairly obvious. We know how to do it, so we just give them a script and say, “Here, just do this.” So that’s what we’ve done, and it works extremely well. So that’s what I sent to the folks at Embry-Riddle. How do you do a TSP project, and that’s what I had done. When I developed the process, I developed all the forms, the form instructions and the scripts for every step of what they had to do.
It took me several hundred hours of work. I mean,
it’s a lot of work, and it is non-trivial. There’s a big, big pile of stuff. When
I got back from
And this is sort of what you’d expect programmers to do. But they got a little bit along like that and they realized they were totally lost. They had no idea where they were. And that, by the way, is where most programmers are most of the time. They don’t know where they are on the project and what to do next. So they basically said, “Okay, we better go back.” And so they did. They basically said, “Okay, we are going to follow the process.” So they did, and they did a marvelous job. They went through and they produced a system. Unfortunately, it never got put in production because one thing or another happened over there. It was a system to manage the flight line at Embry-Riddle. They were going to develop a flight and instructor scheduling system, and they put together the requirements. They went through the design. They did the whole thing. But the process basically gave them an anchor, so they knew where they were, what they should do next.
It was quite helpful,
but they had a problem getting started; their plan wasn’t very good; lots of
stuff. It was pretty obvious from that that they needed guidance when they were
starting the project. We couldn’t just send it to them and say, “Go start.” We decided
to call it a “launch” of a project. And we needed to have somebody to coach the
project. It occurred to me -- I talked about my wrestling coach -- really high
performance teams need coaching. So we decided we were going to have TSP
coaches. We originally thought of them as coaches for the launch, so they would
basically launch the project, and then the team would just go do it. So we went
and did a couple of industry teams. I think we did a couple up at Harris, up in
I kept running into
this, by the way, in organizations. You get halfway through and they get a
re-organization, and you get a new manager. The continuity in business is so
appalling that you literally can’t keep stuff going. In most of the places
where we worked -- a fairly high percentage -- you get something started and
then the sponsor moves on and something else happens, and it’s dead. It’s gone.
You’ve wasted an enormous amount of effort. And it happens all the time. It’s
just astounding. And it can’t be just us. It’s got to happen on everything
going on in industry. So it’s kind of frustrating. I don’t know if it’s just
The Teradyne Team Launch
But in any event, I
was the launch coach for a number of projects that we launched. I remember
doing one at Teradyne out in
And they were
basically trying to find line failures, and it was for the telcos
The opening meeting is with senior management. We bring in marketing people who will describe what the customer wants, the customer situation. And the reason for this is interesting and that is that a lot of the decisions you make when you're producing a system are tradeoffs. And they're tradeoffs of what you can do technically and what the business can afford and what the customer wants. And so the people who are really making those tradeoffs in very many cases are the programmers. Nobody else is involved in enough of the detail to understand it. So the programmers are making these very sophisticated business tradeoffs without the vaguest idea of what's going on in the business.
So we start the launch by bringing in management to give the team the business perspective and bringing in the marketing or customer people. And we bring in customer people if we can, who will tell them what they want and why. And then the programmers have a perspective for making decisions. And what's interesting is during the launch we open the launch with management and then we basically close the door and have a meeting with the team. And I'll come back to that one on a later team launch we had with Boeing, which was fascinating.
But in any event we close the door, and the team goes off and puts together the plan. So we went through the opening meeting with this team at Teradyne, and the management was absolutely firm it had to have the product in nine months. I poked at that a little bit. In the opening meeting the team is supposed to ask questions but they practically never do. The team leader may ask a couple, but by and large management says what they want and nobody says “boo.” So I asked a couple of sort of quick questions. I didn't want to say too much because you don't want to tell management that they're crazy and that sort of thing – at least not until you have a plan.
The team doesn't know enough to argue yet. So basically the team is in listening mode and the management was really very firm. When the meeting broke I went out to the washroom and then I joined them in the meeting room and they were in turmoil. They were irate. You have heard of the storming phase? Well they were storming. They were saying, "This is impossible. We can't possibly do it in nine months. This is crazy. The last project took two years and it was a disaster and this is more complicated than that."
And so I asked them I said, "Whose date is the nine months?" They said, "It's their date, management’s date." I said, "Okay. So what do you want to do?" And they said, "We ought to go back and tell them it's crazy." I said, "If you do what will happen?" They said, "Well they'll beat us up and we'll, you know." they kind of mumbled around and they finally concluded, "Yeah we'll end up that we will try if you insist but it's a very tight date we don't think that we can make it."
I said, "Okay if you do that now who owns the nine months?" They said, "Oh we do." And I said, "Do you want to own the nine months?" And they said, "No." I said, "Okay. So here's what you can do". I said, "You've got to make a plan and do your utmost to make a plan that will end in nine months. If you can't do that then you'll know why and you'll have various alternatives you can say with some evidence that you can do the following. But at least you'll be able to understand in some detail exactly what it'll take to do the job and you'll have a foundation for debating the subject with your management." So they bought it. So we went and made a plan.
And the launch process goes through starting there. We later added stuff to the launch, but we didn't do all of this at the Teradyne because we kept learning as we went ahead. But the process fundamentally is, we come out of the management meeting, you go into a meeting on basically goals and roles. What is it we're trying to do and how are we going to divvy up the work among team members? Now this wasn't divvying up the project work; it's divvying up the management of the job.
And so the TSP actually has role managers, and so we have a planning manager and a design manager and a quality manager and a process manger and a test manager et cetera. And these are people who have specific responsibilities for aspects of the job. The test manager, for instance, doesn’t run all the testing. But the test manager is the person, if there is any issue with testing, you give it to the test manager, and he or she will take care of it. Make sure the test tools are ready in time, that they're thinking about testing when they're doing requirements work, et cetera. Same on design: what methods are we going to use? Make sure everyone is using the same design approach and deciding what is the notation we are going to use and what are our standards? The quality manager is looking at the quality data with the process manager and deciding how to do inspections and this sort of thing. So these are the jobs these role managers do. And what's interesting is that on most projects today, no one does them. They're all left for the project leader to handle. And the project leaders are handling all these mechanics. They're trying to track the plan and this other stuff and they don’t do it very well. They don’t have time.
And so fundamentally most of the stuff that these role managers do doesn’t get done on a typical project. And so the basic rule we use on the TSP is that team leader's job is to make sure every definable task is given to somebody on the team. The team leader's job is to do all the indefinable tasks. He or she handles the motivation, the tracking, protecting the team from management. Staffing is a constant problem -- the minute you launch a team, management wants to borrow somebody. And you've seen that before.
Booch: Oh yes.
Humphrey: And so the team leader's job is to protect the team to make sure issues get handled, to work on risks, and I mean a whole bunch of stuff and motivation and measurement, evaluation, process discipline, reporting. Team leaders are very busy people and they are the ones who actually make the difference between whether the project succeeds or not -- the team leader and the coach. What's interesting -- you think about like a ball team that is in the cellar, who do you get rid of to fix the team? You don’t get rid of the players; you replace the coach or the manager. And we don't normally have coaches in the software community. Coaches are extraordinarily important.
This is one of the issues I keep running into -- what's the difference between a team leader and a coach? Why can't the team leaders be coaches? Well, we're probably going to have to come to more manager-coaches just because coaches get laid off and team leaders don't. We're running into that now in the in the economic squeeze. Unfortunately, management doesn’t recognize the enormous value of coaches.
Booch: This reminds me of the story that you told me earlier in your wrestling career -- how under the one Olympics quality coach you guys performed very differently than with the subsequent coach.
Humphrey: Exactly. Exactly. And we don't see that in the software… We don't see it with development teams at all. And I've not so far talked about my experience with my early teams in engineering. When I was managing development teams I was really acting more like a coach. If you remember, I was asking people what they were doing and focusing on why you're doing that and this sort of thing. I wasn’t really saying do this, do that, do that. We had a lot of things that we had to do, and we had to put together a plan, but I was amazed at how motivating that was. People love it and it worked like a dream. And it worked extremely well here in Teradyne.
So to continue with the Teradyne team: First they went through goals and roles and set that up, and then we get into a strategy session where we figure out what's the process we're going to use. What's the strategy -- well we do the strategy first. What's the strategy for developing this product? And people aren't sure what the strategy is, but there is an enormous range of strategies you can have as you well know. And that is, you can build the whole thing in one big bang, you can decide to build versions of it. You can use all kinds of cyclic processes, you can prototype stuff. There are lots of different ways to build a system. And you may not make every part following the same strategy; you may have several strategies depending on what are the tough problems You want to identify the real nuts that will have complex technical issues and get them on the table early. And maybe you want to prototype them. All kinds of stuff. And so they work out the strategy first. You have quite a bit of discussion on that. What is it? How are you going to do this and why? And you think about the risks and what are the problems and exposures you've got.
So we think through alternatives -- you may want to just build it the way you did the last one. And then once they lay out a strategy then we say, "Okay well now what's the process you're going to use?" And notice we have all these unit processes that they've got. They know how to do configuration management work, how you write modulus and test them and all that sort of stuff. But then you have to weave that together into an overall process that you can use for actually making a plan. And so we have the team do that before they make a plan. And the purpose of this is to make sure you don't confuse people, you don't confuse how to do the job with how long it'll take. Because when you start right away to make a plan and you're thinking about how you're going to finish it in time while you're planning. It's amazing how often people came up and say we don't have time to do that.
And the contention that we follow with the TSP is that you want to do it the right way, and the issue is to figure out what the right way is and lay out your plan based on that. And what you'll discover is if you’re really doing it the right way it is the fastest, and cheapest and highest quality way to do it. And so we have the team start by figuring out what's the right way to do it, and only then do they estimate how long it'll take and the resources. And that turns out to be a very powerful way to do it because now the team has something to defend that they believe it.
And so the team has a big debate about the process flow and how do you want to do that? Do you really want to do this? Do we want to inspect every module? And the guys, you know, once they've got data and quality information they realize: yes, we're going to inspect every module and yes, we're going to do this and we're going to do that, et cetera. And you get the teams to buy that they're going to have personal reviews and that sort of thing. So they went through all of that and I talked about goals earlier, the goals they put together; management goals and team goals.
And in the second meeting, they actually set goals for the yields (% of defects found) they want to find in their reviews and what they're going to accomplish in terms of various things. There are a whole series of goals that they've already made when they're going through a lot of this planning, and so when they finish that they finally have got a process defined, they've got a list of all the products they've got the build. They put all that together and then they make a team plan. And so in the next meeting, the whole team works through how big are the parts, how long will they take to do each step, and they lay that out. They're not talking about who's going to do any of the steps yet. They're just laying out the whole job, and how much time it'll take, and what the effort is, and they also lay out how much time the individuals are going to have.