Home > Articles > Software Development & Management > Management: Lifecycle, Project, Team

Project Planning for Successful Software Development

The software industry may never mature into the lucrative and diverse industry that we would all like it to become. But that doesn't have to be the future. Tyson Gill tells you why we should be focusing on the production of a simple but effective software blueprint during the project-planning phase.
This is an excerpt from Tyson Gill's book, Planning Smarter: Creating Blueprint-Quality Software Specifications.
This chapter is from the book

We've got big trouble in Software City. In the famous musical The Music Man, a fast-talking salesman comes to a small town named River City, Iowa. He tells the gullible folks that they "got big trouble." He warns them that the new pool hall will induce their children to fall prey to bad habits. The only thing that will save them, he declares, is to provide a wholesome alternative, such as a boys' marching band. Of course, it just so happens that the salesman deals in a fine line of quality band uniforms and instruments.

Like The Music Man, I'm here to tell you that we got big trouble in Software City. But unlike The Music Man, I'm not saying that just to peddle a textbook equivalent of 76 trombones and 110 cornets. While some authors applaud the state of the industry, I have serious concerns. While there are particular teams that do experience success on certain projects, such successes are not the norm.

Sources such as objective statistics, self-grading, client reaction, and anecdotal experience all confirm that the rate of failed software development projects is embarrassingly high. Failure rates from 80 to 90 percent are commonly cited and generally accepted. Of course, the accuracy of these figures depends on how success or failure is measured, but by any measure, the situation is not good. The need to improve our software development processes is very real.

To make the picture worse, the cited failure rates reflect only projects that overtly failed to meet objectives. They do not include "successful" projects that cost far more than they might have, or took far longer to complete than was necessary. I disagree with authors who claim that failure rates are artificially high because they are based on unrealistic expectations (DeMarco, 1995). On the contrary, the expectations for minimal success across the industry are far too low. Furthermore, success is most often measured by short-term deliverables, not by long-term maintainability. If failure to maintain were also considered, the picture would look even more dismal.

Poor performance is a pervasive and recurring problem that affects the credibility, effectiveness, and long-term survival of the software industry. Despite an undeniable need, fewer potential clients are willing to risk precious capital on custom development projects. They have either heard of, or they have experienced first hand, the development nightmares that all too commonly occur in these projects. As of this writing, the market for custom application development continues to diminish. I don't believe this is due to a lack of need, but rather to a lack of faith in our industry to deliver.

Clients are not the only ones who are disenchanted. Developers are perhaps even more frustrated. Talented developers don't want to waste their talent over and over again on doomed projects. New developers get disillusioned with the business and find other work. Managers burn out from having to fight the same client relationship fires all day long. Owners of development shops sometimes just give up application development and change their business to provide more profitable and less problematic software services.

We face the real possibility that the fledgling software industry will collapse into a much smaller number of players producing mass-market solutions. It may never mature into the lucrative and diverse industry that we would all like it to become.

This should not, and need not, be the future of our industry.1 Custom software development can and should be a thriving, rewarding, and cost-effective industry for both the consumer and for the provider. How is it that an industry that specializes in organizing and streamlining other business practices can itself be so abysmally unorganized and inefficient?

When a problem is identified, our first reaction is to look around for someone or something to blame. Where does the blame lie here? Do we blame the people involved for creating this situation? Have they simply failed to implement the excellent processes and practices that are well-documented in the literature? Many would say yes. However, intelligent, talented, and experienced professionals staff many failed projects. We cannot simply dismiss them all as incompetent. We should search for root causes of the problems, evolve effective countermeasures, and apply them. We should create a continuous process improvement ethic.

Are the "best practices" that we turn to for guidance perhaps not as good as proponents would like us to believe? I am not willing to say that. As I said before, the practices and approaches recommended in the literature are mostly sound, practical, and based on proven experience. We can't minimize their value and effectiveness.

So what is it then? What is out of kilter in the software development industry? In my opinion, the source of the problem is not individuals or any specific practices, but overcomplication. In our conscientious efforts to minimize risk, adopt best practices, meet budget constraints, prevent wasted effort and so on, we introduce overly restrictive and complicated schemes which do not meet the fundamental requirements for productive planning. Our solutions, in part, make worse the very problems we are trying to avoid. We expend far too much time and effort in project planning and management with too little payback.

The key to improved success is to simplify the process. Eliminate unnecessary effort. Expose and eradicate the sources of those problems that force the application of overly complex and counterproductive practices. This is best accomplished by focusing on the production of a simple but effective software blueprint during the project-planning phase.

2.1 The Pathology of Poor Planning

In the human body, certain materials called allergens stimulate allergic responses. The immune system responds by creating antibodies to combat the allergens. As a side effect, the victim can suffer histamine responses like itchiness and a runny nose, or even more severe life-threatening allergic reactions. In a similar fashion, relatively minor problems identified in a software project often stimulate the production of management countermeasures. Too often, these responses cause even worse, and sometimes fatal, reactions in the project.

The allergens in software projects are almost always buggy code and missed deadlines. In effect, they have lower than acceptable quality and productivity. These symptoms may in turn have been caused or aggravated by poor planning. They stimulate typical responses including more elaborate planning and a tightening of restrictions on the developer's ability to improvise. When this becomes a cyclical pattern, projects suffer. Either they become too congested to even breathe, or at the very least they suffer diminished ability to perform at peak capacity.

Software development is uniquely challenging, but it is not rocket science.2 By that I mean that what we do is not that complicated, or at least it should not be. We analyze requirements and translate them into a relatively simple coding language. We're not putting a space shuttle into orbit or producing a blockbuster motion picture. So how can we have so much difficulty? How can we fall prey to the same few common modes of failure over and over again in our industry?

In the human body, the different systems interact intimately. Similarly, planning, development, management, and the client together form the systems of a software project. These systems interact so intimately that any cure must be holistic in nature, considering the response of all systems to stresses or remedies. Each of these areas has a very different perspective on the typical recurring problems we face in software projects.

Management Perspective

From its perspective, management may sometimes recognize that the marketing department overcommitted deliverables. More often, it simply sees that the developers failed to deliver a satisfactory product. Its dissatisfaction may be based upon profitability, budget, timing, functionality, maintainability, usability, or any number of metrics. When this happens, management has limited courses of action available. It can replace or retrain developers if it suspects that the developers lack competence. It can insist upon more planning if it assesses that the planning was lacking. And it can tighten up management controls to ensure that developers follow the plan more closely. Most often, management applies a combination of these remedies in shotgun fashion, hoping to eliminate any problems next time around.

Figure 2-133 Industry insiders are usually the last to recognize and correct their systemic problems.

Planners' Perspective

From the planners' perspective, we see a slight variation. When projects run into trouble, planners also see a fairly limited number of causes. Planners often feel that marketing doomed planning efforts right from the start by prematurely promising features, approaches, and budget. They feel that the planning process was rushed, inadequately budgeted, and impatiently supported by management. Planners feel that the development group did not fully utilize their planning documentation and that management allowed them to deviate too far from the planned specification. The project planner's remedy for these problems almost always involves more planning next time with greater management control over development.

Developers' Perspective

Developers feel they were handed unrealistic deadlines, constraints, and requirements. They feel they were given a poor project plan that did not meet their needs. They feel neglected and ignored by management when they ask for support to augment or adjust the plan, and then they feel hammered upon when deadlines are missed. Often they feel that management undermined development efforts by understaffing or moving critical staff in midproject. Developers typically want software planning, but they want better, not necessarily more, software planning. They want management, but they seek more responsive and supportive management, not more restrictive, counterproductive management.

Clients' Perspective

From the clients' viewpoint, they just want the software done well, and of course completed as quickly and inexpensively as possible. Poor planning is the first thing they blame if there are problems in the project. They usually feel that the planners did not understand their needs thoroughly enough and did not document adequately. They often complain that although they got what they specified, they did not get what was needed.

Sounds kind of like the interaction between parents and their kids, doesn't it? The parents just want their kids to get an "A" in school. The kids may want that also, but when the kids get only a "C," the parents blame themselves for not taking a more active role. They grimly tighten up and create more restrictions for their kids and impose penalties if the kids deviate. They try to gather more measurable data by requesting more frequent grade reports from the teacher. When increased parental management conflicts with other demands upon them, the kids divert energy into working around those constraints and end up getting a "D" next time. Both sides have taken what they consider to be reasonable action at each step, but the interaction still degenerates into an increasingly counterproductive and hostile relationship.

This same sort of situation occurs over and over again in software development. As a consultant, I was sometimes called upon to save failed projects. Management invariably insisted that the reasons for failure lie within planning and development. If they blame themselves, it is because they didn't manage enough. They vow to be more diligent and proactive about increased project management next time.

The developers want both more and less management at the same time. What they really mean, even if they haven't realized it, is that they want more supportive management.

When you talk to the planners, they lament that they didn't have time to do more software planning and puzzle over why the developers have such problems after all the "good planning" they did do.

When I came into these situations, it was usually only after management had already tried replacing developers several times and had tightened up management practices drastically. It still wasn't working, and probably had gotten worse. In some of these cases, management had become so desperate that it was willing to give me a free hand. In those situations, I was able to succeed dramatically.

In other cases, management insisted that I fix the project while adhering to all of the management controls and requirements it had put in place. I declined those projects. I knew there was no way I could succeed where other fine developers had failed if I was locked into the same underlying causes of failure.

Accordingly, for the software planning techniques in this book to succeed, management may need to "loosen up." That does not necessarily mean managing less, but managing differently, managing smarter. This book is called Planning Smarter, not "Managing Smarter," so it is not specifically about project management. However, smarter planning cannot succeed without smarter management, so it will touch on some management issues and practices.

"Poor management can increase software costs more rapidly than any other factor." (Robertson, 1999)4

Planning smarter can indirectly encourage smarter management. When managers and planners cooperate well, successes are transmitted and reinforced throughout the system.

Is the developer the blameless victim in all this? Certainly not! There are times when developers fail to deliver. Yes, there are hack programmers out there. Between teaching, hiring, and consulting, I have seen some amazingly bad ones. The bad practices that programmers can adopt are truly astounding. However, better planning and tighter project management generally do not even begin to fix coding problems. The only things that can mitigate these problems are training and talented, experienced developers with good programming standards and coding reviews. These are the types of problems and solutions that my first book addressed (Gill, 2000).

Quite often, however, the developers are not the problem, or at least not the only problem. No developer, however talented and conscientious, can develop a project efficiently from a bad plan. The rule of "garbage in, garbage out" is universal. Sometimes, through heroic efforts, developers manage to succeed despite working from a nearly useless project plan. However, in many cases a bad plan all but guarantees development problems. A mediocre programmer may be able to succeed if given a good plan, but he has no chance at all working from a bad plan.

"Attracting the best people is vital, but it is also essential to support them with an effectively managed software process." (Humphrey, 1989)5

Certainly, a bad programmer could mess up the best-laid plan, but it is not common to have a great plan and a poor developer. More frequently, you have the worst case of bad planning followed by bad implementation. In this case, the planners usually don't see the true nature of their complicity. Since the programmer obviously failed to achieve his or her goals, management fails to appreciate that the plan might have made success quite difficult. Managers conclude instead that they simply failed to exert enough control over the developers. They resolve to spend more time and money next time to maintain greater oversight and constraints upon the developers, limiting the ability of good developers to succeed despite the bad plan.

"A lot of software projects fail, but we software developers are not such dummies that our sheer incompetence can account for them all." (DeMarco, 1998)6

It is worth emphasizing again that the one thing that management, planning, and development all normally agree on is that more planning is better. More planning is the solution to all problems. It is the panacea of the software industry. When projects run into trouble, management almost invariably concludes that more planning is needed and tighter controls must be put in place to adhere to it. Managers reason that if they had invested more in planning, the good developers could be more efficient and the less experienced developers would encounter fewer problems.

The flaw in this is that it assumes that planning is always good. This mistaken assumption leads to the erroneous conclusion that more planning must be better. The reality is that planning is often quite bad. It can be as appallingly bad as the worst coding samples. When you try to fix the problems caused by bad planning with even more bad planning, the problems compound.

When a project has been poorly planned, it is usually no secret. Everyone, at least all the developers, knows it very early in the project and becomes quickly resigned to yet another Death March even before the ink has dried on the functional specification.

"A Death March is a project that is doomed from the start. Everyone knows it and yet everyone just resigns to march along dutifully to certain doom." (Yourdon, 1999)7

It can quickly reach the point at which no planning may be better than all that bad planning. No matter how skilled the developers may be, the quality of the plan they receive is critical to their success. Typical software planning often fails to provide the developers with what they need to meet expectations. That may seem like a harsh assessment. After all, the industry has almost universally agreed upon the merits of current planning tools and techniques. However, it is also commonly accepted that the majority of software projects fail to meet their goals.

To explain this contradiction, most software professionals would conclude that their peers simply fail to apply well-known and proven planning principles. I am not willing to accept that explanation. My follow-up question is, why do they fail to apply them?

Walk into any software shop and you hear almost everyone there talk about project planning. Everyone appreciates the importance of doing it effectively. Many attempt to apply standard planning methodologies to their projects. They usually fail.

Some talk about it incessantly and never do it. Some read all the literature and continually bemoan the fact that their shop doesn't implement this or that practice. Some produce incredibly complex planning documents that don't seem to provide much benefit.

In my mind, if it isn't getting done well, you must simplify the process rather than make it larger and more complicated. Threatening, educating, and licensing people into compliance isn't going to have much real benefit if the practices are simply too complex or cumbersome to apply effectively. To eliminate the pathology of poor planning, we must simplify project requirements to the essential elements required to specify the desired software. Then we must put processes in place to obtain that information in the most efficient and direct way possible. The first example of a well-known document that is typically overcomplicated and overinflated is the famous Vision Document.

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.

Overview


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information


To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information


Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children


This site is not directed to children under the age of 13.

Marketing


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information


If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information


Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents


California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure


Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact


Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice


We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020