Agile Software Development: Why It's Hot!

By Jim Highsmith

Date: Mar 15, 2002

Article is provided courtesy of Addison Wesley.

Return to the article

Agile is all around us - and it actually has been for some time, as Jim Highsmith points out in this article. So why is it suddenly so blisteringly hot?

In the last year, Agile software development (ASD) has burst upon the development scene, led by the indefatigable XP aficionados. But the Agile pot has been simmering beneath the surface for more than a decade. In the past year, articles on Agile development have appeared in ComputerWorld, Software Development, IEEE Computer and Software, Cutter IT Journal, CIO, and even the Economist. I think Agile's sudden rise to fame is rooted in two facts:

Three key questions in assessing this hotly debated trend in our profession:

The future of our Information Age economy belongs to the Agile—those companies that have the capacity to create change (and maybe even a little chaos) for their competitors. If you can innovate better and faster, you create change for your competitors. If you can respond quickly to competitive initiatives, new technology, and customers' requirements, you create change for competitors. If you are slower, less innovative, less responsive—then you're doomed to survival strategies in a sea of chaos imposed by others. Is your company going to set the pace of change, or are competitors going to set it? In our Information Age economy, a company's ability to set the pace, to create change, lies in its ability to develop software. In a world of constant change, traditional rigorous software development methods are insufficient for success.

What Is Agility?

If turbulence and turmoil define the problem, then agility is key to the solution.

Agility isn't a one-shot deal that can be checked off the organizational initiative list. Agility is a way of life—a constantly emerging and changing response to business turbulence. Critics may counter, "Agility is merely waiting for bad things to happen, and then responding. It's a fancy name for lack of planning and ad hoc–ism." But Agile organizations still plan; they just understand the limits of planning. Three characteristics help define agility:

Agility is the ability to both create and respond to change in order to profit in a turbulent business environment.

Action Over Reaction

Agility is not merely reaction, but also action. First and foremost, Agile organizations create change—change that causes intense pressure on competitors. Creating change requires innovation—the ability to create new knowledge that provides business value. Second, Agile organizations have an ability to react, to respond quickly and effectively to both anticipated and unanticipated changes in the business environment.

Exploration

In our volatile economy, companies need to enhance their "exploration" skills at every level of the organization. Good explorers are Agile explorers—they know how to juggle and improvise. Indiana Jones was a good explorer, somehow living through every outlandish adventure. Agility means quickness, lightness, and nimbleness—the ability to act rapidly, the ability to do the minimum necessary to get a job done, and the ability to adapt to changing conditions. Agility also requires innovation and creativity—the ability to envision new products and new ways of doing business. In particular, IT organizations have not done an adequate job of balancing the needs of exploration and optimization.

Improvisation

Agile individuals can improvise; they know the rules and boundaries, but they also know when the problem at hand has moved into uncharted areas. They know how to extend their knowledge into unforeseen realms, to experiment, and to learn. When critical things need to get done, call on the great improvisers.

Improvisation makes great jazz bands. From a few key structural rules, jazz bands improvise extensively. Having a solid foundation enables their tremendous flexibility without allowing the music to degenerate into chaos. The proponents of business process reengineering and software engineering methodologies probably blanch at the thought that improvisation, rather than carefully articulated processes, is key to success. Yet in today's turbulent environment, staff members with good balancing, judging, and improvisational skills are truly invaluable.

Being Agile means trusting in one's ability to respond more than trusting in one's ability to plan.

What Are Agile Software Development Ecosystems (ASDEs)?

I began writing a book about Agile software development methodologies, but I kept worrying about the word methodology because it didn't fit with the focal points of Agile development—people, relationships, and uncertainty. Furthermore, by using the word methodology, Agile practices are instantly compared to traditional software development methodologies—thereby using the wrong measuring stick for comparison. So I began to use the term Agile software development ecosystem to describe a holistic environment that includes three interwoven components:

I use the term Agilists to identify those who are proponents of ASDEs.

Some people think that Agile means fewer processes, less ceremony, and briefer documents, but it has a much broader perspective, which is the primary reason for using the word ecosystem rather than methodology. Although fewer processes and less formality might lower development costs, they're not enough to produce agility. Focusing on people and their interactions and giving individuals the power to make quick decisions and to self-adapt their own processes are key to Agile ecosystems.

The word ecosystem conjures up a vision of living things and their interactions with each other. Within an organizational context, an ecosystem can then be thought of as a dynamic, ever-changing environment in which people and organizations constantly initiate actions and respond to each other's actions. The word ecosystem focuses us on the dynamic interactions of individuals and teams rather than on the static lines on organization charts.

"Chaordic" Perspective

To fully understand ASDEs, we need to understand each of the three components and how they relate to each other. First, Agilists share a view that organizations are chaordic—that every organization exhibits properties of both chaos and order that defy management through the use of linear, predictive planning and execution practices. Viewing organizations as chaordic means understanding that the predictability upon which traditional project management and development lifecycle practices are built is a root cause of dysfunctionality among customer, management, and development organizations. A chaordic perspective affects both how we respond to change and how we manage project teams and organizations.

In day-to-day project work, a chaordic perspective creates two outcomes that are 180 degrees out of sync with rigorous methodologies:

Although ASDEs involve careful planning, the fundamental assumption remains that in a turbulent environment, plans are not predictable—at least at the level of project scope, schedule, and cost. Plans are hypotheses to be tested rather than predictions to be realized. However, the product goals of the business are achievable, in large part because Agile people adapt. They can "adapt" to an articulated vision and a schedule, scope, or cost goal through tradeoffs in the other two dimensions. And while process can aid people in working together, in volatile environments the idea of driving out process variation through measurement and correction—statistical process control—becomes an unworkable hypothesis. Changes that are the result of knowledge gained during the project—knowledge not discernable early in the project—require processes that can respond to change, not ones that attempt to eliminate it.

Peter Senge uses the term mental model to identify the perspective, set of assumptions, stories, and beliefs that each of us carries in our mind that provide a context for thinking (The Fifth Discipline: The Art and Practice of the Learning Organization, Currency/Doubleday, 1994). In organizations, the collective set of mental models defines an overall cultural context. Companies that are heavily sales-oriented differ from those that are heavily engineering-oriented. Companies whose driving strategy is customer intimacy differ from those whose driving force is product innovation. Companies whose mental model includes linearity, cause and effect, hierarchy, predictability, and control will operate very differently from those whose mental models include collaborative networks, emergence, decentralization of power, and acceptance of unpredictability. One is Newtonian, the other chaordic.

Collaborative Values and Principles

The second piece of the interconnected web that defines ASDEs is the statement of collaborative values and principles. While it's difficult to characterize the Agile Manifesto in one word, collaborative seems to be the best single adjective. Values and principles shape the ecosystem. Without a set of stated values and principles, an ecosystem is sterile, reflecting practices but not the people who interact within it.

A collaborative culture includes people and their relationships within a development team and with customers, management, and partnering teams within or external to their own company. Human dynamics, communications, and collaboration may be known as the "soft" sciences, but in practice, they may be the hardest to master. Principles and values help define a culture—the environment in which people want to work.

Barely Sufficient Methodology

The final component of an ASDE is methodology. The traditional definition of methodology includes things such as roles, activities, processes, techniques, and tools. Alistair Cockburn summarizes these components when he defines methodology as "the conventions we agree to"—the ways in which people work together on a project (Agile Software Development, Addison-Wesley, 2001, ISBN 0-201-69969-9).

In The Social Life of Information, John Seely Brown and Paul Duguid (Harvard Business School Press, 2000) discuss the major differences between process (as used by the business process reengineering movement) and practice:

The ASDE model provides a practice-centered approach to methodology rather than a process-centered approach.

There are two reasons to pursue barely sufficient methodologies: value and innovation.

Methodology also relates to organizational model. Agile methodologies contain minimal processes and documentation and reduced ceremony (formality). Agile methodologies are labeled "barely sufficient" (Cockburn) or "a little bit less than just enough" (Highsmith, Adaptive Software Development: A Collaborative Approach to Managing Complex Systems [Dorset House, 2000]), or "minimal" (Bob Charette's Lean Development Guidelines). However, this streamlining of methodology isn't just based on reducing work effort, but more importantly is based on understanding the chaordic worldview—one in which emergent (innovative) results are best generated at the "edge of chaos," perched midway between chaos and order.

Practices (or techniques) are the lifeblood of methodology. Whether it's pair programming, Scrum meetings, customer focus groups, or automated testing, when carried out by talented and skilled individuals the practices of ASDEs produce results.

What Kinds of Problems Does Agility Solve Best?

In recent years, software technology has moved from supporting business operations to becoming a critical component of business strategy. It drives new product development, from automobiles with hundreds of chips with embedded software to cellular phones and other wireless devices that are extending the definition of "distributed" systems.

Agile software development (ASD) is tuned to innovation and response—to creating new knowledge that delivers value to businesses and to responding quickly to competitive challenges. Rigorous software methodologies (RSMs) are useful, but for a set of problem domains that is shrinking. Many of the techniques from RSMs can be effectively employed by ASD approaches, but the framework and philosophy of the two are different. Agile approaches are best employed to explore new ground and to power teams for which innovation and creativity are paramount.

People have focused on the title rather than the subtitle of Kent Beck's groundbreaking book, Extreme Programming Explained: Embrace Change (Addison-Wesley, 2000, ISBN 0-201-61641-6). These individuals, especially managers in large organizations, are put off by the word extreme and its connotation of daredevils swooping in on their coding snowboards. Or, if they manage to force themselves past the word extreme, they land on the word programming and relegate the material to that "mechanical" stuff that the geeky people do. Although the words in Kent's book may talk about programming, and he may even advocate such extreme practices as testing one's own code, the strategic issue surrounding XP, and all other ASDEs, concerns embracing change.

Agile organizations create chaos for their competitors, first by creating change so fast that competitors are left gasping for breath, and second by responding quickly to competitors' attempts to change the market. Just imagine what it feels like to have a competitor with a new product-development cycle of 12 months to your 18; furthermore, every time you introduce a breakthrough feature, they match it in their next product release. They're attacking; you're always on the defensive. That's the essence of Agile organizations—creating change that you can live with and your competition can't. "The source of our competitiveness in this industry is our ability to manage in a chaotic environment," says Silicon Graphics' CEO Ed McCracken. "But it's more proactive than that. We actually help create chaos in the first place—that's what keeps a lot of potential competitors out." (See Marco Iansiti, Technology Integration: Making Critical Choices in a Dynamic World, Harvard Business School Press, 1997.)

Agile organizations don't just respond to change; they generate it!

What Is the Future of Agile Software Development?

So, what about the future? To the extent that the future business environment continues to be turbulent, I think rigorous cultures face a difficult challenge. No amount of process thinning or document pruning will make them Agile—Agile is an attitude, a sense of how the world works in complex ways. However, to the extent that executives and managers still want the world to be predictable and plan-able, Agile cultures and ASDEs will be difficult to implement. In the final analysis, businesses gradually but inevitably gravitate to practices that make them successful, and ASDEs will increasingly contribute to successful software projects. Rigorous approaches will remain, but there will be many fewer companies using them five years from now.