Dismantle the Waterfall by Using Integrated Teams
Although incremental development is the way to go, you don't want to make the mistake of having a whole series of little waterfalls. Although this is slightly better than one big waterfall, it misses the point. Software development is not a mechanical task. Isolated specialties fail at software development because of the wicked problems.
Software development is a social, learning activity. This is why a solo developer can outperform a team on small tasks. When programming-in-the-small, one person can do the analysis, design, programming, and testingshe understands the entire context and doesn't suffer from the inefficiencies of handoffs between the various roles.
Integrated teams can get the same benefits. Rather than use specialists, make sure that your developers are involved in all of the activities. The analyst should work closely with the designers, programmers, and testers. The analyst should even write some of the code and do hands-on testing of the application. Only by staying current with all of the lifecycle activities can the analyst respond to the various issues as they arise.
Similarly, your designers should be deeply involved in the programming tasks. Jim Coplien calls this the Architect Also Implements pattern. It provides the designer with direct feedback about the quality of the design as well as making it much easier for the designer to explain the design issues to the programmers.
The programmers and testers need to be involved in the analysis and design activities. This ensures that they gain the deep domain knowledge necessary for understanding the big picture.
Cross-functional Teams Are the Way to Go
My preference is to label everyone in the team a "developer." Sure, some will be better at analysis than others, and some will be better at design or programming. The key, though, is to make sure that the team doesn't waste time doing costly handoffs inside the team. Instead, whoever does the analysis work should sit down with the people who will be doing the design, programming, and testing and jointly come up with a design that can be implemented and tested. By continually revisiting these "design" sessions as the implementation proceeds, a cross-functional team can ensure that even the wicked problems get handled smoothly.
Obviously, it would be great to go further and to include some users as part of each cross-functional team. That would ensure that even emergent requirements could get handled smoothly.
Incremental Development Revisited
With a cross-functional team, progress is measured using binary milestones. A requirement or feature is either implemented and tested or it isn't. You might think that this would mean too much time elapsing between milestones, but that turns out not to be the case.
Your cross-functional team will have about five people in it. This allows for one analyst, one designer, two programmers, and one tester, but all of these developers are cross-trained to be able to support each other's specialties. Assuming that you're using use cases, your requirements can be fairly granular, typically taking 515 developer weeks to implement and test. This means that every one to three calendar weeks each cross-functional team will deliver additional functionality.
By ganging up on the requirements, the team makes it possible to measure progress fairly effectively. Estimating to create an overall project plan is easy because all you have to do is classify each use case as simple, medium, or complex. On the last day of each increment, while the rest of the team is finishing off the testing, the analyst-developer for the next use case can meet with the users to get more details about the use case. When the rest of the team is finished with the prior use case, they can then meet and refine the overall estimate for the new use case. This keeps the overall project plan intact and allows the project manager to make reasonable predictions about when the functionality will be complete.
By using an integrated, cross-functional team, we can dismantle the waterfall and regain control of our projects.