Unsustainable Software Development and its Causes
Date: Nov 11, 2005
Sample Chapter is provided courtesy of Addison-Wesley Professional.
Unsustainable development, as depicted in Figure 2-1, is an all-too common situation today in the software industry. Most software teams place too much of a short-term emphasis on feature development and fixing defects and do not pay enough attention to the health of the underlying software. The result is software with a high cost of change, that is increasingly unmaintainable, and where every change has the risk of destabilizing the product.
)
Figure 2-1 Unsustainable development is characterized by a constantly increasing cost of change to the software. The usual evidence of a high cost of change is a constantly increasing number of defects. Each change adds complexity and uncovers or causes defects that require more changes, and this complexity leads to a declining ability to respond to customer requests and changes to the ecosystem.
In unsustainable development, teams tend to spend an ever-increasing amount of time fixing defects and stabilizing the software. Features still get developed, but less time is available for feature development due to the amount of time required to stabilize the software, which increases with each new release. This results in teams tending to become change-adverse while they are stabilizing the product because each change increases the risk that something will break. Teams tend to desire freezing of requirements as early as possible so they can get some work completed. This reduces their ability to respond to customer requests and changes to underlying hardware or software because they are spending too much time on stabilization, which is wasted effort, and not enough on new work.
The symptoms of a short-term outlook are only evident once a threshold has been crossed. This is the point where the development team starts to spend so much time trying to keep up with the defect backlog that they are noticeably unable to spend enough time to develop new features. It may take only a few months or many years to reach this point. Many development teams may not even be aware how bad things are: Release cycles might become death marches, customers, management, and sales force complaints might get louder, and developers themselves will start blaming the problems on management ("if only they had listened to me...") and other developers ("they produce too much buggy code..."). It may take a while, but morale will eventually suffer then plummet; in extreme cases, people will lose their jobs through being fired, layoffs, or outsourcing, and customers will lose patience with the product and company.
Luckily, the extreme is seldom reached. Unluckily, most project teams live in a gray area that is not sustainable development but is uncomfortably close to unsustainability. These projects have not reached the threshold where the problem is obvious; developers feel like they are barely keeping up or are making slow progress but with too much stress. This is the software development death spiral, where the only variable between unsustainable projects is the rapidity of the descent toward obvious unsustainability.
Technical Debt and the Flywheel
The underlying cause of the inability to develop new features due to a defect burden is what is best called technical debt. Technical debt is caused by an accumulation of poor decisions over time—decisions that often seem right at the time but are usually made in the interests of a short-term quick fix to the software. Very rarely is it possible to identify a single decision that stands out as one where the problems might have started.
Jim Collins describes a relevant flywheel metaphor [Collins 2001]. Collins uses the flywheel metaphor to describe how decisions affect momentum in an organization. Think of a massive flywheel that is so large all you can do at first is give it tiny nudges that move it an inch at a time. Your organization’s sustainable development emphasis is like continually providing tiny nudges. Each new nudge adds momentum to the flywheel so that it continues to pick up speed. However, every decision to develop a new feature or fix a bug by introducing an ugly hack or ignoring the underlying architecture acts as a brake on the flywheel. The software development death spiral results because these brakes, or nudges in the wrong direction, will eventually stop or even get the flywheel spinning backwards. How fast the flywheel is spinning and its direction is something your organization should be aware of.
The Perils of Jumping in Place
One strong piece of evidence for technical debt in the software industry is the number of products that must be completely rebuilt in order to keep up with competitive products or to respond to a major technology change. Completely rewriting a product requires a massive investment in effort and time, and the result is most often a loss of market share while the next generation (replacement) product is built. These companies are building new products to jump in place, to address the same market with a similar product! Even if the product is truly better, the loss of market share and customer mindshare is not easily overcome, especially since existing competitors will have gotten stronger and new competitors entered the market during the time required to build the next generation product.
An extreme example of losing market share when jumping in place is best dubbed the Osborne effect after Osborne Computer, Inc. Osborne produced one of the first portable computers in the 1980s. It was the size of a small suitcase and could barely fit under an airplane seat, yet it revolutionized personal computers at the time because it came preinstalled with useful software and was portable to some degree, something that is taken for granted today, largely thanks to Osborne. Not many users took advantage of the portability of the computers due to their size and weight, but it didn’t matter because of the value of the system and its preinstalled software. Osborne had a successful computer on the market that was selling well. Then, for some reason, it decided to build a replacement rather than refining its current product. Osborne compounded its mistake by preannouncing the next generation of its computer with a larger screen, faster processor, etc. This was a huge mistake; customers stopped buying Osborne computers to wait for the new generation, cash got tight, the new product fell behind schedule, and soon Osborne Computer was in Chapter 11 bankruptcy.
Bankruptcy is the extreme end of a spectrum of lost market share. Lost market share in any form requires effort to get back to the former position, and this is wasted effort that technology companies must be diligent to avoid.
The Causes of Unsustainable Development
Why is unsustainable development so common in the software industry? It’s because software development is a complex undertaking. Figure 2-2depicts some of the factors that contribute to the complexity. There are a number of project stresses, which you can’t control, that are trying to push your project toward unsustainability. Your only defense is to employ a number of things that you can control, or at least influence, to counteract the stresses and keep the project on a sustainable path. Get one or more of these wrong and your project is going to start into the death spiral. The project stresses are going to be different for every project, and over time the stresses are bound to change. Therefore, constant diligence is required on your part to keep the balance.
)
Figure 2-2 Software projects are subjected to a number of project stresses that can’t be controlled. These stresses are a constantly changing set of forces that push your project towards unsustainability. You can employ a number of defenses that you can control or influence to try and keep the project on a sustainable path.
Project Stresses
User Requirements
User requirements continually evolve and change. It is virtually impossible to predict what users are going to require more than some small number of weeks to months in the future.
External Dependencies
Today’s software depends on operating systems, libraries, third-party components, etc. Software usually also depends on computer hardware or hardware interfaces (especially if it’s embedded). The steady advance of technology means that changes to these interfaces are a given. Managing these changes is a challenge and requires constant diligence and foresight. A change to any of these might set a software team back for days, weeks, or months as the new version of the operating system, compiler, device driver, libraries, web browser component, etc. is integrated and tested. And for many software teams, especially where it is impossible to dictate the end user’s computing environment, the team must support the old and the new interfaces. This adds to development complexity and to the risk that the software will break in unexpected ways.
Competition
It is relatively easy to set up a software organization today because the barriers to entry are low. All that is needed are some bright people, a few computers, and a connection to the Internet to get started.
Unfortunately, what many companies lack is a workable business plan. They do not have a clear idea of their true market worth because they overestimate or do not know their market size, and they almost always suffer from an inability to adequately sell and market their work. They may be good at getting initial funding and making some initial sales, but most software organizations struggle to stay in business. Their very existence and sometimes more innovative products make it difficult for other companies to thrive.
Disruptive Technologies
The constant advance of technology means there is a need for continual innovation, or at least a need to keep up. Given that products need to meet users’ demands over the long term, it is tempting to think that in every case it makes perfect sense to always do only what your customer wants you to do. In many cases, however, this may actually be a long-term recipe for disaster. Because of the constant advance of technology, it is possible for disruptive technologies and associated ways of doing business to eliminate established businesses, businesses that in most cases have done an excellent job of listening to their customers! [Christensen 2003] [Christensen and Raynor 2003] The problem is that most customers are not visionaries and are restricted by their current problems. Hence, they are really good at asking for 10% more than what they currently have, but if another solution, usually built on newer technology, comes along that offers them 50% more in one increment, chances are they’ll take it.
Disruptive technologies usually start out looking like a toy to established businesses and their customers. Because they do not appear to be a threat at first or require a new way of doing business, the established businesses will ignore them right up until the time when their customers realize the advantages of the new offerings and decide to change. By then, of course, it’s too late for the established business.
The Internet is an example of a disruptive technology. The Internet is a godsend in terms of providing a ubiquitous communication medium. Software and related services such as support and consulting can cheaply and easily be offered over the Internet. All businesses today must learn how to wisely use the Internet. They must also learn how to deal with the threats of the Internet, because it also provides an easily accessible means for hackers to exploit an internal network, for illicit software and serial numbers to be distributed, and also a forum for malicious people to exploit security flaws in software and distribute those exploits to unsuspecting users.
Disruptive Business Models
Equally disruptive to most organizations is a disruptive business model. Dell Computers is an example of this type of disruption. Dell does not innovate technologically; their innovation is the ability to produce computers on demand by clustering their suppliers together and taking all orders over the Internet. This gives them the ability to compete on price and responsiveness. Established PC manufacturers such as HP, Compaq, and IBM have not been able to keep pace or duplicate Dell’s business.
Another example of a disruptive business model is open source software. The open source community is dedicated to producing software for free. Although virtually all of this software is only usable by highly technical people (typically other programmers) today, there is no question that open source will play an increasingly disruptive role in the software industry as usability plays an increasingly important role in open source projects. Microsoft has a well-established monopoly on the desktop market that is now being eroded in some sectors, such as in governments, because Microsoft cannot compete against free software. However, what remains to be seen with open source projects such as Linux is if the business model is sustainable, whether the open source community can keep itself going almost exclusively through support and services as claimed.
I believe it is important for all software developers and organizations to embrace open source. As I will argue later in this book, it is important that the software community stop repeatedly implementing commodity software components so that everyone can concentrate on their unique value. Open source is an ideal mechanism to make this happen. In addition, I believe the tools used for software development have not kept pace with the demands of modern software development. Many organizations and teams put a great deal of effort into building their own infrastructure (custom-built tools, bug tracking, and configuration management systems, etc.) when this type of infrastructure should be shared.
Cost Management
Organizations need to carefully manage how much they spend on soft- ware projects. Because of the complexity of software development, it is hard to control costs through greater efficiency. Achieving high efficiency is difficult because of factors such as the mindset of the organization, its processes, training (which does not emphasize efficiency), deadlines, etc. Hence, the easiest way to reduce costs is to employ cheaper people. This explains the increasing use of outsourcing of software development to countries such as India and China where labor is cheaper. But outsourcing is the easy way out. More people in the industry need to recognize the need for efficiency, because efficiency is at the heart of sustainable de- velopment.
Project Controls
Collaboration
Working effectively with other people in your organization and your customers is a crucial aspect of software development. If people are off doing their own thing, then the project as a whole suffers. Good things happen when people can cooperatively work together toward a common objective.
Methodology
The development methodology, the principles (mindset) and practices (project planning, design, testing, coding, etc.) you use, has a dramatic impact on the success of a project. Methodology directly influences efficiency, change tolerance, responsiveness, and software quality.
As described in the Introduction, an important factor is that we use the wrong metaphor for software development (that software development is like building a building). Because the metaphor is incorrect, teams either approach software development using plan-driven development, because that is what is taught to them in school as the approach used by engineers, or,if that doesn’t work, they use variants of an ad-hoc code-then-fix approach.
Software teams who approach software development with little or no discipline commonly employ code-then-fix. Sometimes this is a reaction to practices that do not work for their project, and sometimes it is a lack of focus on technical excellence. These teams spend most of their time writing software and very little time on planning and analysis. Software developed this way is fragile and brittle and will resist change because even if good coding practices are followed, the lack of a set of sound practices will lead to a great deal of dead code and thrashing.
Plan-driven development is derived from engineering as it is applied to the design and construction of bridges and buildings. It is best typified by the ISO-9000 set of standards. The plan-driven approach is what is taught in schools to software developers because there is the mistaken belief that all the complexities in software and the target ecosystem can be understood before any code is written. This is rarely true over the short term and definitely false over the longer term.
Much of the emphasis in plan-driven development is on producing documentation of requirements, designs, test plans, etc. While documentation has value as an important means of communication, the main point is missed: that the real value comes from the process you have to go through to produce the documentation, not from the documentation itself. That is, having to work through a design with your peers is a valuable exercise because it fosters collaboration, understanding, and creativity; producing the document that describes the results of the discussion is less valuable, and the actual document is of even lower value because it must be maintained. By stressing the need to create documentation, plan-driven approaches achieve a mixed result, where on the positive side, stressing documents means that analysis work is required, but the result is too often that the effort spent producing and maintaining the documents detracts from the amount of time spent on analysis. Also, the emphasis on documentation has led many people to believe that the existence of the documents equates with discipline, when in fact discipline should be viewed as ensuring that the analysis work happens in the most efficient manner possible.
Plan-driven development also does not work for software development because it is impossible to put together a plan when the project starts that take into account all the changes in requirements and in the ecosystem that will occur over the course of the project. Also, plan-driven approaches stifle agility because they emphasize ceremony and documentation over having a working product in the erroneous belief that somehow having a pile of documentation describing requirements and design is proof that the project cannot fail, that a disciplined approach is being used, and that by extension any project that does not have this depth of documentation is not disciplined. The problem is that the effort put into the ceremony and documentation is wasted effort; many projects only wind up producing documentation when what they really needed was to get a working product into their user’s hands and get feedback from them since that is the only way to really know if the desired system meets user needs.
Expertise
Expertise with software development and with the project’s ecosystem is a strong indicator of success. Expertise allows teams to apply an increasing amount of intuition and to be able to understand what is important and what is not.
Decision Making
Your ability to make decisions in a complex and constantly evolving ecosystem, often with incomplete information, is crucial to success. An important aspect of decision making is prioritization, where you choose what must be worked on and when.
Leadership
Leadership helps set and communicate a vision, strategy, and tactics. Leadership helps to keep projects on track.
Culture
The culture of an organization shapes the attitudes and priorities of the people who work in it. Culture defines the requirements for leaders, how decision making is performed (and by whom), and the methodology. If a new methodology or leadership or decision-making style is required, then the culture must change.
Simplicity and Reliability
The vast majority of software today is overly complex, hard to use and learn, and has too many features for most of its users. Even commonplace applications like word processors are packed with features that most users never use. For example, one study showed that although the Office97 version of Microsoft Word had 265 functions, only 12 functions were used by more than 75% of users, and 42 were not used at all! [McGrenere 2000]
There have not been enough studies done to prove any ill effects of unnecessary complexity on users. However, I believe this is a significant issue as computing struggles to become more widely adopted, not to mention provide the productivity gains that should come through automation. Complexity is behind at least part of the outsourcing trend because one way to manage complexity is to get someone else to do it for you. Complexity is also why my wife asked me to return a recent Christmas present: I bought some software that claims to help process and manage digital pictures, yet after one look, she said the new software undeniably had some useful features but overall it would mean she would spend more time processing pictures than before.
When it comes to computing and simplicity, there is a huge disparity between what consumers want and what companies would like to sell them [Economist 2004]. Too many software organizations still compete on features and price alone. Unfortunately, they are often encouraged to do so by the most technical users, who are the early adopters of any technology, and by the media, who still publish reviews that contain feature comparison charts.
Related to simplicity is the need for reliability. Software is notoriously unreliable. Many users refuse to buy the first version of any software product, and users are frequently frustrated by version upgrades that break or remove features they relied upon in the previous version. Too many software organizations emphasize features over reliability.
There is not enough emphasis placed on simplicity and reliability. And yet, all one needs to do is look at Apple’s iPod or a Palm PDA. These are simple and reliable devices that are popular because they do a few things very well. What is impressive about them is that a great deal of restraint was shown in their development. It’s easy to add features but very hard to show restraint to keep them out. Is there a future where this type of thinking is applied to software?
Summary
Unsustainable development is a development pace that is typified by stress, frustration, and a sense of not being in control. It is most evidenced by a continually increasing cost of change and defect rate and a corresponding decreasing ability to respond to changing conditions. Unfortunately, unsustainable development encompasses a large gray area of declining capability with a variable pace of decline. As a consequence, unsustainable development may not be immediately evident, even to the people on the project.
Unsustainable development results from a complex ecosystem of continual change and stresses that are largely out of the control of an organization. Therefore, in order to achieve sustainability, the organization must recognize its stresses and continually adjust its defenses and coping mechanisms.
The next chapter returns to the theme of sustainability. It starts by describing the difference between principles and practices and then outlines some of the key principles that lead to sustainability. These principles are used to frame the practices described in the rest of the book, because sustainable development begins with principles (mindset) and uses practices that reinforce them.