Home > Articles > Software Development & Management

This chapter is from the book

Problems with Programs

The gap between analytics and operational or transactional systems explored in the previous section explains why organizations have such a hard time deriving insight from data and applying that insight to their operational systems. A second major flaw with current approaches is problems with programs. Fundamentally, the programs used to run most organizations today are built without any embedded intelligence. The "peopleware" of an organization, not the software, is what makes decisions.

The inherent inefficiency of these programs has long been obvious to those working with information systems. Attempts have been made to address this limitation. All these attempts, from handwritten code to failed attempts at artificial intelligence, from enterprise applications to business process management and service-oriented architecture, have made little difference. Most organizations have software programs that just aren't smart enough.

The Weight of Legacy Code

As more custom code is written, organizations find that it lives longer than they intended. Code that's 20 or even 30 years old is still in use in many organizations—still running core business processes and processing vital transactions. Companies have discovered the hard way that they can't code their way into the future. This deadweight of legacy applications has created two problems.

Part of the problem comes from a mind-set that systems, like other enterprise assets, should be built to last. This focus results in detailed but largely static requirements and huge investments in system architecture and design. However, it also buries critical code in complex systems. To make applications robust and complete, a huge amount of business expertise must be embedded in the system, but there are problems with this approach:

  • Embedding business expertise in the system is hard because those who understand the business can't code, and those who understand the code don't run the business. Business users can't explain to programmers what they need, and the result is systems that don't quite work the way they were intended.
  • Generally, custom code isn't well documented, or the documentation is allowed to get out of date rapidly. The promises of "self-documenting" code notwithstanding, new generations of programmers struggle to amend code to face new challenges.
  • Showing a regulator or auditor how custom code works is nearly impossible, and demonstrating compliance with policies or regulations is extremely difficult, costly, and time consuming.
  • New challenges and requirements emerge constantly because the world doesn't stop changing, and organizations can't afford to stand still. Therefore, changing and managing custom code is difficult.

All these factors contribute to another aspect of the problem: the maintenance backlog. The maintenance backlog is the list of projects not progressing because of a lack of time and resources or other projects having higher priorities. Most projects don't even make it to the backlog unless they have a positive potential—that is, the project's business value exceeds its cost. An organization that could magically complete all projects in its backlog would add tremendous value to the business. For most organizations, this backlog represents a sink of resources and time that could add significant value.

Organizations have a maintenance backlog for many reasons, but one of the most persistent is that a huge percentage of IT resources is spent on systems maintenance—75 percent or more, as noted at the beginning of the chapter. So much old code is used to run businesses and must be constantly updated (to reflect new regulations, competitors, and products) that this work dominates the IT department's responsibilities. The systems were originally built to specification but no longer do what the business needs. Perhaps the specification was wrong, or perhaps the business has changed. Maintenance takes so long and uses so many resources that little or nothing else can be done.

Even if maintenance work isn't consuming a large percentage of your IT resources, traditional approaches to embedding logic in systems create rigid and unwieldy systems. This lack of agility causes problems if you need to respond quickly and cost-effectively to a competitive issue, new regulations, a new channel, or another major change. Coding business logic into legacy systems perpetuates the separation between those who know the business and those who run the systems and makes it hard to update systems as business needs change. Expert systems and 4GLs were two programming approaches intended to address these issues.

Artificial Intelligence and Expert Systems

In the 1980s, the IT industry and organizations made a major investment in various forms of artificial intelligence (AI), which was designed to bring the power of human intelligence to computers. Expert systems vendors promised that their software would perform tasks just like the company's most experienced employees. These vendors used intelligence and best practices painstakingly collected from industry experts to power their systems, but most expert systems didn't succeed in practical application.

Expert systems were typically designed as "closed systems," designed to solve a problem on their own. They didn't support or integrate with the programming models prevalent in OLTP systems. Specialized hardware and software were required. At the time, most corporate computer systems were written in COBOL and ran on IBM mainframes, but many expert systems packages required high-end workstations using artificial intelligence languages, such as Lisp or PROLOG. Organizations couldn't easily integrate these systems into their production environments and had no personnel trained in maintenance or programming techniques for them. They had to rely on special training and support from software vendors. Generally, expert systems came with predefined rules for accomplishing specific tasks. Specialist programmers at vendors, working from interpretations of interviews with industry experts, crafted these rules as compromises between the different methods their sources used.

Organizations purchasing expert systems software usually needed to go through laborious tuning sessions to understand how the rules functioned and to modify them for their business preferences. Organizations found it impossible to use expert systems to automate other tasks because they couldn't modify their underlying processing flow and structure. In the end, the organizations that experimented with expert systems became wary of computer software promising "intelligent processing."

In addition, AI software consumed huge amounts of computing resources at a time when these resources were still at a premium. Other factors prevailed as well, especially rampant paranoia that Japan's burgeoning economy would overwhelm the United States and that Japan's government-funded "fifth-generation computing" initiative, largely about developing AI, would pose a bigger threat to the U.S. economy than all the Toyotas it could produce. This mentality created a rush among small AI entrepreneurs to market immature and nonperforming products. Ultimately, the Japanese fifth-generation initiative failed to produce much more than factory (and toy) robots, and many AI vendors returned to the university labs and defense contracts from whence they came. To be fair, some firms survived and, having learned a painful lesson, transformed their products into more commercially useful offerings, such as business rules management systems, data-mining tools, logistics optimization software, and embedded intelligence, such as fraud detection.

Ultimately, even though the outcome was less than desirable, everyone learned from the experience. Lessons included the value of keeping knowledge (rules) in a repository where it could be managed and the power of a declarative approach, compared to procedural programming, to simplify some problems. Expectations for AI were rolled back to more reasonable levels, the buying community became more careful about the next big trend, and AI entrepreneurs learned that they have to embed their inventions in useful applications as well as cooperative architectures. In the meantime, the power and relative cost of processing is dramatically more favorable, and open standards and ubiquitous communication provided the basis for AI-like technologies to have a second chance.

4GLs and Other User-Friendly Tools

If business know-how is hard to embed into information systems, even with an expert system, can you make maintaining the code easier? To bring a higher level of abstraction to programming problems, 4GLs were developed. In theory, abstraction makes it easier to see what's happening in code, engages businesspeople in the process more effectively, and makes IT development and modification of sophisticated processes easier and quicker. Many 4GLs came and went, and not much changed in the problems of application maintenance.

Although 4GLs do offer some productivity gains and many are easy for less technical staff to use, they fail to address the core "build to last" problems. Code representing core business logic is still procedural and embedded in code that's "plumbing" or otherwise highly technical. Few business users become fluent enough to write code themselves, and those who do often create programs and scripts that aren't managed or controlled well enough for a company to rely on.

Stretching the metaphor, you could consider a spreadsheet a 4GL (at least its scripting or macro creation capabilities). The problem of maintaining spreadsheet applications is well documented and pervasive. Names such as shadow IT, spreadmarts, and spreadsheet hell are used. The problem is that spreadsheets are helpful tools for composing a piece of analysis, but as shared applications, they are a disaster. They lack version control, a repository, and collaboration features. They are consistently applied to problems for which they were never intended, which is their biggest weakness—in fact, the biggest weakness of almost any piece of technology.

Business Rules

The use of business rules as a way to specify how an organization behaves began to gain ground in the mid-1990s and was popularized by Ron Ross4 and Barbara von Halle,5 among others. Many early adopters regarded business rules primarily as a tool for describing and understanding how an organization wants to act. In one sense, business rules were a way to design an organization. As technology support for this approach increased, business rules began to separate into a business-design approach and a higher-level, more declarative way to develop code. Declarative approaches allow managing each piece of code or logic separately instead of in the procedural sequence typical of regular code.

Another major impetus for the business rules approach was to bring the idea of business rule management to problems that didn't involve extremely complex decisions (such as diagnosing cancer). Instead, this approach was used with everyday operational transactions that occur in high volume and involve decisions with low to medium complexity. Instead of using an expert system to handle very complex problems, organizations could use business rules to automate 80 percent of less complex cases and, therefore, improve throughput. Additionally, this approach enabled nontechnical businesspeople to state business rules rather than provide fuzzy requirements that IT translates into the system's real logic—sometimes without enough business input and often with a lot of confusion.

Despite the early and repeated proof of the effectiveness of business rules, their use has been somewhat limited. Most organizations using this approach at a strategic level do so only to describe and understand their business. Organizations using it to make specific systems smarter often do so only in a localized way. The rules are extracted from limited sources and largely ignore the insight that can be gained from an organization's data. Without an overall approach, few organizations have become proactive at finding decisions and automating them with business rules, although this approach is clearly possible. Business rules, as you'll see, are a critical component of a solution to the problem of making systems smart enough, but their potential has been untapped so far.

Buying Solutions

In the past, IT departments had no alternative but to build their own software. As the industry matured, however, more packaged applications became available that offered quicker time to market and less need for specialist programmers. In theory, you could buy a package for inventory control, for example, and install it and be up and running quickly.

In fact, many packages were overly rigid, based on a single interpretation of how a certain business process might run. Configuring and installing these packages, especially as they grew into today's enterprise applications, were time consuming and costly. In addition, enterprise applications still assumed that people were the motivating force behind systems. They had a data model, captured data through various generations of user interface technology, and stored it in an operational database. They provided reporting or integrated with business intelligence/data warehouse products so that data could be transferred to an analytic environment and used to help people make decisions. Attempts to customize or extend these applications resulted in custom code with all the problems described previously. IT departments still spent a lot of money and time on maintenance, organizations still had backlogs, and systems still didn't do what businesses needed them to do .

Processes and Services

At the end of the twentieth century, a new class of software became perceived as a way to solve many problems with IT: workflow or business process management (BPM) software. These products generated high initial ROI because they integrated many disparate systems, linked people in different departments into a coherent process, and made management and reporting of an overall business process possible, often for the first time.

BPM systems, however, still assumed that intelligence and decision making in processes come from people or are embedded in systems. The use of worklists, alerts, and the paraphernalia of integrating people into processes is widespread. Most BPM systems allow limited replacement of human decision making with automated decision making. Those that do tend to focus on routing and other kinds of simple decision making, not making decisions about operational transactions.

In parallel with the growth of business process management, service-oriented architecture (SOA) started making inroads in IT departments. SOA promised a new level of agility and flexibility and reduced maintenance backlogs. To some extent, it's delivering on these promises. However, an SOA approach doesn't change how business expertise is turned into computer code, nor does it address the issues in delivering analytic data discussed previously.

Although BPM systems and SOA don't offer a way to make systems (or even processes) smart enough for today's business environment, they do offer a framework for using the approaches and technologies discussed in this book. Integrating business rules engines and analytics with the process automation and orchestration capabilities they offer can make smart enough systems more possible.

One common factor in problems with different approaches to programs is people. Relying on people to make decisions has a consequence: You can move only at the speed of people.

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


This site is not directed to children under the age of 13.


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020