Introduction to Agile Concepts in Real-Time and Embedded Systems Development

This chapter is from the book

Real-Time Agility: The Harmony/ESW Method for Real-Time and Embedded Systems Development

Learn More Buy

This chapter is from the book

This chapter is from the book 

Real-Time Agility: The Harmony/ESW Method for Real-Time and Embedded Systems Development

Learn More Buy

Benefits of Agile Methods

The primary goal of an agile project is to develop working software that meets the needs of the stakeholders. It isn’t to produce documentation (although documentation will be part of the delivered system). It isn’t to attend meetings (although meetings will be held). It isn’t to create schedules (but a schedule is a critical planning tool for all agile projects). It isn’t to create productivity metrics (although they will help the team identify problems and barriers to success).¹³ You may do all of these things during the pursuit of your primary goal, but it is key to remember that those activities are secondary and performed only as a means of achieving your primary goal. Too often, both managers and developers forget this and lose focus. Many projects spend significant effort without even bothering to assess whether that effort aids in the pursuit of the development of the software.

The second most important goal of an agile project is to enable follow-on software development. This means that the previously developed software must have an architecture that enables the next set of features or extensions, documentation so that the follow-on team can understand and modify that software, support to understand and manage the risks of the development, and an infrastructure for change and configuration management (CM).

The benefits of agile methods usually discussed are:

• Rapid learning about the project requirements and technologies used to realize them
• Early return on investment (ROI)
• Satisfied stakeholders
• Increased control
• Responsiveness to change
• Earlier and greater reduction in project risk¹⁴
• Efficient high-quality development

These are real, if sometimes intangible, benefits that properly applied agile methods bring to the project, the developers, their company, the customer, and the ultimate user.

Rapid Learning

Rapid learning means that the development team learns about the project earlier because they are paying attention. Specifically, agile methods focus on early feedback, enabling dynamic planning. This is in contrast to traditional approaches that involve ballistic planning. Ballistic planning is all done up front with the expectation that physics will guide the (silver) bullet unerringly to its target (see Figure 1.4). Agile’s dynamic planning can be thought of as “planning to replan.” It’s not that agile developers don’t make plans; it’s just that they don’t believe their own marketing hype and are willing to improve their plans as more information becomes available.

Figure 1.4 Ballistic versus dynamic planning

Since software development is relatively unpredictable, ballistic planning, for all its popularity, is infeasible. The advantage of early feedback is that it enables dynamic planning. A Law of Douglass¹⁵ is “The more you know, the more you know.” This perhaps obvious syllogism means that as you work through the project, you learn. This deeper understanding of the project enables more accurate predictions about when the project will be complete and the effort the project will require. As shown in Figure 1.5, the ongoing course corrections result in decreasing the zone of uncertainty.

Figure 1.5 Reduction in uncertainty

Early Return on Investment

Early return on investment means that with an agile approach, partial functionality is provided far sooner than in a traditional waterfall process. The latter delivers all-or-none functionality at the end point, and the former delivers incremental functionality frequently throughout the duration of the development. As you can see in Figure 1.6, agile delivers high value early, with less incremental value as the system becomes increasingly complete, whereas the waterfall process delivers nothing until the end.

Figure 1.6 Percent value returned over time

Another way to view this is by looking at incremental value over time, as shown in Figure 1.7. We see that an agile process delivers increasing value over time, whereas the waterfall process delivers no value until the end.

Figure 1.7 Incremental value

Delivering value early is good for a couple of reasons. First, if the funding is removed or the project must end early, something of value exists at the point of termination. This is not true for the waterfall process, but it is a primary value in an agile process. Additionally, delivering validated, if partial, functionality early reduces risk, as we see in Figure 1.8. Exactly how early deliveries do this is a topic we will discuss in more detail later, but let us say for now that because we validate each incremental build and we tend to do high-risk things early, we significantly and quickly reduce the project risk. The waterfall process reduces risk slowly at first because you only really know about the quality and correctness of things that you validate, and validation comes only at the end in a waterfall process.

Figure 1.8 Risk over time

How can we return incremental value for a system that is delivered externally, such as a cell phone or a missile? Every increment period (which the Harmony/ESW¹⁶ process refers to as a microcycle), a system is designed, implemented, and validated in accordance with its mission statement. This mission statement identifies the functionality, target platform, architectural intent, and defect repairs to be included. The incremental functionality is organized around a small set of use cases running on an identified (but not necessarily final) target environment. Through the use of good engineering practice, we can encapsulate away the platform details and ensure that the delivered functionality is correct, given the current target. For example, for one tracking system, our team originally targeted laptops with simulated radars and created actual validated functionality on that environment. Over the course of the project, as hardware became available, we migrated to target hardware of the actual military systems. Through this approach, we had high-quality, testable software earlier than expected.

Satisfied Stakeholders

Stakeholders are simply people who have a stake in the successful outcome of a project. Projects have all kinds of stakeholders. Customers and marketers are focused on the functional benefits of the system and are willing to invest real money to make it happen. Their focus is on specifying to the developers the needs of the users in a realistic and effective fashion. Managers are stakeholders who manage that (real or potential) investment for the company to achieve a timely, cost-effective delivery of said functionality. Their job is to plan and schedule the project so that it can be produced to satisfy the customer and meet the users’ needs. The users are the ones who use the system in their work environment and need high-quality functionality that enables their workflow to be correct, accurate, and efficient. All these stakeholders care about the product but differ in the focus of their concern. The customers care how much they pay for the system and the degree to which it improves the users’ work. The managers primarily care how much the system costs to develop and how long that effort takes. The users primarily care about the (positive) impact the system makes on their work.

Agile methods provide early visibility to validated functionality. This functionality can be demonstrated to the stakeholders and even delivered. This is in stark contrast to traditional preliminary design review (PDR) and critical design review (CDR) milestones in which text is delivered that describes promised functionality in technological terms. Customers can—and should—be involved in reviewing the functionality of the validated incremental versions of the system. Indeed, the functionality can be implemented using a number of different strategies, depending on what the process optimization criterion is. Possible criteria include the following:

• Highest-risk first
• Most critical first
• Infrastructure first
• Available information first

All other things being equal, we prefer to deliver high-risk first, because this optimizes early risk reduction. However, if the users are to deploy early versions of the system, then criticality-first makes more sense. In some cases, we deploy architectural infrastructure early to enable more complex functionality or product variations. And sometimes we don’t have all the necessary information at our fingertips before we must begin, so the things we don’t know can be put off until the necessary information becomes available.

Improved Control

Many, if not most, software projects are out of control, to some degree or another. This is largely because although projects are planned in detail, they aren’t tracked with any rigor. Even for those projects that are tracked, tracking is usually done on the wrong things, such as SLOC delivered. Thus most projects are either not tracked or track the wrong project properties.

Project tracking requires the answers to three questions:

• Why track?
• What should be tracked?
• How should projects be tracked?

Why track? Project teams that don’t know exactly why they are tracking project properties rarely do a good job. Only by identifying the goals of tracking can you decide what measures should be tracked and how to implement the tracking procedures.

The biggest single reason for project tracking is that plans are always made in the presence of incomplete knowledge and are therefore inaccurate to some degree. Tracking enables the project deviance from plan to be identified early enough to effectively do something about it. Projects should be tracked so that they can be effectively managed, replanned as appropriate, and even scrapped if necessary. You can effectively replan only when you know more than you did when the original plan was made, and that information can come from tracking the right things. Put another way, the fundamental purpose of tracking is to reduce uncertainty and thereby improve project control.

What should be tracked? Ideally, tracking should directly reduce uncertainty in the key project characteristics that relate to the cost, time, effort, and quality of the product; that is, tracking should directly measure cost, time to completion, effort to completion, and defect rates. The problem is that these quantities are not directly measurable.

So projects typically evaluate metrics that are measurable with the expectation that they correlate with the desired project quantities. Hence, people measure properties such as lines of code or defects repaired. The flaw in those measures is that they do not correlate strongly with the project criteria. If, at the end of the project, you remove lines of code during optimization, are you performing negative work and reducing the time, cost, or effort? If I don’t know exactly how many lines of code I’m going to end up with, what does writing another 10,000 lines of code mean in terms of percent completeness? If I measure cyclomatic complexity, am I demonstrating that the system is correct? The answer is an emphatic no.

The problem with many of the common metrics is that while they are easy to measure, they don’t correlate well with the desired information. This is because those metrics track against the project implementation rather than the project goal. If you want to measure completeness, measure the number of requirements validated, not the number of lines of code written. If you want to measure quality, measure defect rates, not cyclomatic complexity. The other measures do add incremental value, but the project team needs to focus on achievement of the ultimate goal, not weak correlates.

Agile methods provide the best metrics of all—working, validated functionality—and they provide those metrics early and often. Agile focuses on delivering correct functionality constantly, providing natural metrics as to the quality and completeness of the system over time. This in turn provides improved project control because true problems become visible much earlier and in a much more precise fashion.

Responsiveness to Change

Life happens, often in ways that directly conflict with our opinions about how it ought to happen. We make plans using the best available knowledge, but that knowledge is imprecise and incomplete and in some cases just wrong. The imprecision means that small incremental errors due to fuzziness in the data can add up to huge errors by the end of the project—the so-called butterfly effect in chaos theory.¹⁷ Chaos theory is little more than the statement that most systems are actually nonlinear; by nonlinear we mean that small causes generate effects that are not proportional to their size. That sums up software development in a nutshell: a highly nonlinear transfer function of user needs into executable software.

The incompleteness problem means that not only do we not know things very precisely, but some things we don’t know at all. I remember one project in which I was working on a handheld pacemaker program meant to be used by physicians to monitor and configure cardiac pacemakers. It was a based on a Z-80-based embedded microcomputer with a very nice form factor and touch screen. The early devices from the Japanese manufacturer provided a BIOS to form the basis of the computing environment. However, once the project began and plans were all made, it became apparent that the BIOS would have to be rewritten for a variety of technically inobvious reasons. Documentation for the BIOS was available from the manufacturer—but only in Japanese. The technical support staff was based in Tokyo and spoke only—you guessed it—Japanese. This little bit of missing information put the project months behind schedule because we had to reverse-engineer the documentation from decompiling the BIOS. It wouldn’t be so bad if that was the only time issues like that came up, but such things seem to come up in every project. There’s always something that wasn’t planned on—a manufacturer canceling a design, a tool vendor going out of business, a key person being attracted away by the competition, a change in company focus, defects in an existing product sucking up all the development resources, . . . the list goes on and on.

Worst, in some way, is that knowledge you have about which you are both convinced and incorrect. This can be as varied as delivery dates, effort to perform tasks, and availability of target platforms. We all make assumptions, and the law of averages dictates that when we make 100 guesses, each of which is 90% certain, 10 are still likely to be wrong.

Despite these effects of nonlinearity, incompleteness, and incorrectness, we still have to develop systems to meet the stakeholders’ needs at a cost they’re willing to pay within the time frames that meet the company’s schedules. So in spite of the nonlinearity, we do our best to plan projects as accurately as possible. And how well do we do that? The answer, from an industry standpoint, is “not very well at all.”

The alternative to plan-and-pray is to plan-track-replan. Agile methods accept that development plans are wrong at some level and that you’ll need to adjust them. Agile methods provide a framework in which you can capture the change, adjust the plans, and redirect the project at a minimal cost and effort. The particular agile approach outlined in this book, known as the Harmony/ESW process, deals with work at three levels of abstraction, as shown in Figure 1.9.

Figure 1.9 Harmony/ESW timescales

The smallest timescale, known as the nanocycle, is about creation in the hour-to-day time frame. In the nanocycle, the developer works off of the work items list, performs small incremental tasks, and verifies that they were done properly via execution. In this time frame, small changes with local scope can be effectively dealt with in the context of a few minutes or hours.

The middle time frame is called the microcycle and focuses on the development of a single integrated validated build of the system with specified functionality. The microcycle time frame is on the order of four to six weeks and delivers formally validated, although perhaps limited, functionality. Changes with medium scope are dealt with in the formal increment review¹⁸ and in the prototype mission statement that identifies the scope for the microcycle iteration.

The largest time frame is called the macrocycle. The macrocycle concerns itself with the beginning and end of the project and primary milestones within that context. The macrocycle is usually 12 to 24 months long and represents a final, or at least significant, customer delivery. At this scope, large-scale changes are managed that may result in significant project replanning.

Earlier and Greater Reduction in Project Risk

The last of the benefits we will discuss in this section has to do with reduction of project risks. In my experience, the leading cause of project failure is simply ignoring risk. Risk is unavoidable, and attempts to ignore it are rarely successful. I am reminded of a company I consulted to that wanted help. The development staff of this medical device company had been working 55 to 60 hours per week for 10 years and had never made a project deadline. They asked that I come and see if I could identify why they were having such problems. As it happens, they did develop high-quality machines but at a higher-than-desirable development cost and in a longer-than-desirable time frame. They consistently ignored risks and had a (informal) policy of refusing to learn from their mistakes. For example, they had a history of projects for fairly similar devices, and it had always taken them five months to validate the machines. However, they, just as always, scheduled one month for validation. They refused to look at why projects were late and adjust future plans to be more reasonable.

In this context, risk means the same thing as it did in the earlier discussion of safety. It is the product of the severity of an undesirable situation and its likelihood. For a project, it is undesirable to be late or over budget or to have critical defects. We can reduce project risks by managing them. We manage them by identifying the key project risks and their properties so that we can reduce them. Risks are managed in a document called either a risk list or a risk management plan. As we will learn later, this risk list contains an ordered list of conditions, severities, likelihoods, and corrective actions known as risk mitigation activities (RMAs). These activities are scheduled into the iterations primarily in order of degree of risk (highest-risk first).

For example, if the risk is that CORBA¹⁹ is too slow to handle the throughput required, an early prototype²⁰ should include some high-bandwidth data exchange and the performance can be measured. If it is found that CORBA does, in fact, provide inadequate performance, other technical solutions can be explored. Because the problem was discovered early, the amount of rework in that case will be less than in a traditional “Oh, God, I hope this works” development approach. In agile methods this kind of an experiment is known as a spike.²¹

The risk list is a dynamic document that is reviewed at least every iteration (during the party phase²²). It is updated as risks are reduced, mitigated, or discovered. Because we’re focusing attention on risk, we can head off an undesirable situation before it surprises us.

Efficient High-Quality Development

High quality is achieved by the proper application of agile methods but in a different way from traditional industrial processes. This is again a dynamic, rather than a ballistic, approach. Agile achieves high quality through continuous execution, continuous integration, and continuous testing—begun as early as possible. Agile holds that the best way not to have defects in a system is not to systematically test them out but to not introduce them into the software in the first place (a topic I will address in more detail in upcoming chapters).

Efficiency is why most people in my experience turn to agile methods. In fact, agile methods have sometimes been thought to sacrifice quality and correctness in the pursuit of development efficiency. It is true that agile methods are a response to so-called heavyweight processes that emphasize paper analysis and ballistic planning over early execution and risk reduction. Nevertheless, agile emphasizes efficiency because it is a universal truth that software costs too much to develop and takes too long. A good agile process is as efficient as possible while achieving the necessary functionality and quality. Agile often recommends lighter-weight approaches to achieve a process workflow.