Introduction to Context-Driven Design
This book is about how to design IT applications. I have written the book because I want to persuade people to design differently. In particular, I want to change application design by
- Basing it on the recognition that you don’t gather your IT application requirements, you design them
- Making it more like an engineering discipline, in particular, by analyzing designs and looking for flaws before implementation
- Ensuring that the application works with other applications that exist or are in development to create a coherent IT architecture
This book is about how you think about application design and how you analyze designs. It says little about how to structure your development team, how you manage the team, and how frequently you give your end users a new version of the application. If you follow the precepts in this book, you can structure and manage your project in many different ways, and there is no right way that suits every organization and every kind of project.
I start by examining the first of these points: designing requirements. This is followed by a section called “What Is Design?” This is preparation for a discussion on the second point above which is in a section entitled “Making IT Application Development More of an Engineering Discipline.” The third point is addressed in “Taking IT Architecture into Account.”
Designing requirements—surely that must be wrong, isn’t it? For design in general, perhaps, but for design of IT applications, no, it is not. The reason is simple. IT applications in business don’t exist in a vacuum; they are there to support the business. Thus the design of the IT application should reflect the design of the business solution, just as the design of a house’s foundations reflects the design of the whole house.
Several chapters of this book are about replacing a passive exercise of gathering requirements for an IT application with an active exercise of designing a business solution supported by an IT application. Let me explain why this is important.
Implementing business change is hard. There are some business changes that executive management can force on the business by moving around money and resources; that is the easy way to make change happen. The difficult way is making existing operations more error free, less costly, and more flexible, in other words, changing what employees are doing and consequently moving them out of their comfort zone. Add an IT application into the mix and it becomes harder still. If the IT application is late, it can delay the introduction of the new way of working. If the IT application is clumsy, unreliable, or slow, it can engender resentment against the whole program of change. If the IT application’s functionality is not what people expected, they will start to question the basis of the program for change.
One of the root causes of these dangers is that when an IT application is involved, the desire for business change must be converted into precisely defined demands for people to do their work differently. But even with the same goal in mind, different people will have different ideas on the best way to achieve a goal. When you add competing departments, a changing business environment, and differing practices and traditions in different geographies, differing visions and conflicts are almost inevitable. With other approaches to business change, like training, you can skate over these differences, sweep them under the carpet as we say, but with a new IT application you can’t. IT applications demand precision; that is the nature of computing and programming. Let us look at a very simple example. You will no doubt have come across a Web site that will accept your input only if you put data in fields marked by an asterisk. Someone somewhere has made the decision that the data is so important (and so unlikely to be faked) that it is worth their while to force potential customers to fill in these fields in spite of the annoyance it causes. Does everyone in the organization agree with the decision on how much data is required? I doubt it. IT applications are full of decisions like this. Some can be easily changed late in the day, like this example of required fields. But some are more structural to the application, and some are structural to how the business is done and cannot easily be changed.
Let us look at a more complex example. I live and work in England, and here in the last few years the banks have been handed down large fines for mis-selling, for abusing the ignorance and naïveté of their customers to sell them inappropriate financial products such as overly expensive or unnecessary Payment Protection Insurance (PPI) that insures against not being able to repay a loan. Our example is an imaginary bank that wants to prevent these practices. First, they have to understand what went wrong; after all, no one instructed people to sell inappropriate products. Mis-selling happened because the bank put pressure on sales through targets and bonuses to sell, sell, sell. The pressure was not only direct but also indirect through the refurbishment of bank branches to make them selling spaces and training that told sales to attach a PPI sale with every loan. Looking forward, the bank still wants to sell, but it wants to do it more subtly, and with more understanding of the customer’s wants and needs. The bank is not going to succeed in making its employees work differently by just telling them to be nicer. Sure, it can put some retraining in place, but all the good words are likely to be forgotten the next time bonus plans are announced or the bank starts a drive for greater efficiency. So how can the bank make the message stick? Eventually the bank decides to make the employees ask key questions during an interview to validate the sale and to record the answers. The questions are designed to ensure that the customer needs the product. It’s not long before someone suggests that the answers to the questions be put into a new database so that if there is a later problem, there is traceability of who said what to whom. Not much longer after that it is realized that the answers to these questions provide valuable marketing information. Shortly after, somebody else (perhaps a golf buddy of the CEO who happens to work for a big consultancy firm) will say that it is really simple to write an application that captures the answers to a bunch of questions and puts them in a database, and it should be done in the twinkling of an eye at practically no cost. However, think about it a bit longer and there are many questions, such as these:
- To what financial products does this apply—all of them?
- If the sales people try to sell to the same customer twice, do they annoy the customer by asking the same set of questions the second time?
- How much time should be spent asking questions rather than selling, and what is the cost to the bank?
- Does the application take input from existing data in the bank? If so, what data and where is it?
- How and who should be allowed to change the questions?
- If the questions are changed, how does this affect the analysis of the data (e.g., analysis that spans the time before and after the change)?
- For some products, such as a mortgage for a house, sales have been asking lots of questions for a long time, so is this a new system, or is it an extension of an existing system; and if it is a new system, how do we prevent the same questions from being asked all over again?
- If the bank hears of a customer’s change of circumstance (such as a new job, moving, a divorce), does the bank go back and reassess the products sold to that customer?
And so on. It’s easy to see how asking many questions could degenerate into laborious bureaucracy to the annoyance of customers and sales staff alike.
In addition to these questions around the process of selling, there are many questions about the IT aspects of the development, such as these:
- To what degree will the new application be integrated with existing applications or databases?
- How should the application be integrated with existing security systems?
- How can we ensure that sales people aren’t given carte blanche to access anyone’s personal financial details?
- Are there any existing tools for analyzing the data?
And so on.
Today in IT application development the approaches to gathering requirements fall into two camps. In one, a team asks the stakeholders questions, singly or in groups, written or verbally, perhaps by having brainstorming sessions or other techniques for developing business ideas. The team then writes a document that specifies the requirements. When the requirements document is reviewed and signed off, it is passed to an IT development department for implementation. In the other school of thought, a short list of pithy statements of the requirements is written, and then the programmers start development. The software is developed in small increments, and detailed requirements are taken on board by talking to the business representative just before the development of each increment. There is also continual feedback on the product as it is produced, expanding the list of pithy statements.
Whichever way the requirements are gleaned, both approaches rely on a set of assumptions that, if not true, can lead the project to go dangerously awry.
The first assumption is that business managers will give clear answers to all questions posed to them. Look back at the list of questions in our bank example. Many managers simply won’t have thought about these questions when the IT requirements are being set. Even worse, some of them will pretend to answer the questions by giving vague, waffly answers, probably with an upbeat, sales-y tone that attempts to push all difficult questions under the carpet. When you have worked gathering requirements for even a short period, you realize that it requires a fanatical attention to detail. Many business managers simply aren’t details people, and they find being pressed on detailed questions uncomfortable.
There are some people who insist that all requirements are measureable, partly to counteract management waffle. For instance, instead of saying, “The application must be easy enough to be used by the general public,” they say things like “Users must complete the task within five minutes and abort the task less than 5% of the time.” Setting metrics is hard. Is a 5% abort rate a good measure of ease of use? Should it be 1% or 10%? And, of course, people abort for reasons other than ease of use, so it is not clear what the metric means. The other problem with such metrics is that they increase costs. Someone has to write extra code in the application to capture the raw information. Someone else has to analyze the data, and if the application fails the test, the project has to be delayed while the problem is fixed—if it can be fixed. And if the abort rate is 6%, what is the business going to do—cancel the application? Sometimes a better approach is to set a requirement to gather the metrics and set the targets as an aspiration, not a showstopper. But there is no point doing even this unless someone is prepared to look at the metrics and do something about them. Business managers will rarely give you target metrics, especially for something as nebulous as ease of use. When they do, they are likely to get it wrong by asking for something that is either far too easy or far too hard. This is where professionalism comes in; the person gathering the requirements should know what is reasonable and help guide the interviewee.
The second assumption is that all stakeholders give consistent answers. This simply isn’t true. Managers disagree with managers. Managers disagree with workers. Headquarters disagrees with the branch or department managers. In our example, different managers are likely to say that different people will set the questions. Perhaps the head of the sales force demands that sales managers can override the questions, changing them for their local offices. The head of marketing might completely disagree.
The third assumption is that all the important stakeholders will be found. Not only do requirements gatherers often leave out important stakeholders, especially, for instance, stakeholders who are based abroad, but sometimes they are instructed to leave out important stakeholders. In our example bank, the central management might anticipate pushback from the branches about having to spend time asking questions the customers don’t want to answer so hope to present them with a fait accompli rather than address the issue up front.
There are cultural differences here. I once heard of a Japanese business manager who characterized the Western approach to business with a gunslinger analogy as “Fast draw, slow bullet.” In other words, Western businesses get a product to market fast, but the details and the backup aren’t there to make it a success. Furthermore, someone is given the role of hero or fall guy, and the rest of the corporation stands back and watches from the sidelines rather than falling in line and giving support. This is a real danger for IT application development as only one person or a few people are providing answers when a wide range of views is necessary. (On the other hand, the problem with many non-Western cultures is that no one tells the manager that he—it is usually a “he”—is wrong.)
The temptation for the person or team gathering the requirements is to answer the requirements questions themselves. It is easy to do this unconsciously, to over-interpret what you are being told or to listen to what is being said with your mind already made up. It is common that if you are sure of the direction in which the project should be going, you only hear statements that confirm your view. Psychologists call this “confirmation bias.”
This leads to the fourth assumption: that business executives and managers will read and understand the requirements specification and give intelligent and well-considered feedback. For projects where feedback is given after looking at an early version of the finished product, confirmation bias works the other way around; managers assume the application will support what they want it to do even if they don’t see it in front of them—they assume it is in another part of the application or that they have misinterpreted how the applications works.
Any business project must be assessed by comparing the cost and time to deliver against the benefits. The fifth assumption is that the requirements team is in a position to give a good estimate of the application development and runtime costs and that senior managers have a sufficient understanding of the functionality proposed that they can make good trade-off decisions.
The final assumption I discuss has to do with the IT department. In many proposed applications these days—and the bank example we have been discussing is typical—there is a question about the degree of integration with other IT applications. Sometimes more integration means more up-front cost, though often with gains later on because some features are built that future development can use. In our bank example, the integration could range from using a common single sign-on for security to integrating the data with existing customer tables in the database. In the latter case, the designer may have to decide which of several customer tables to integrate with. For instance, there may be customer tables in the accounts database, the mortgage database, and the insurance database. (In real life, I expect most banks have many more than three customer tables.) The assumption, therefore, is that the degree of integration and the pros and cons of different options will be sufficiently visible to business managers that they can make a considered decision.
On a small project with a small number of stakeholders who work well together and where there is little or no integration with other IT applications, these assumptions may well be true. But as projects increase in size and integration becomes more important, these assumptions are far less likely to be true. The consequence is mainly that the IT application does not meet the business need. It commonly happens that someone notices that the project won’t do what he or she thought it would do, and hence the requirements change mid-project, leading to delays and cost overruns. (As I said in the preface, it is commonly said that rapid business change is the reason for requirements changing, but I think that by far the most common reason is a realization by the stakeholders that the project is going in a direction they never wanted.) Sometimes the requirements gatherers themselves realize these assumptions are wrong and react by making the requirements overblown because they are trying to include all possible business practices.
This book is about a different way to engage with the stakeholders to build requirements. It is based on a simple observation, which is that the requirements for an IT application are the output from a design process. Put another way, you cannot just “gather” requirements for an IT application like picking apples from a tree. You have to design the business solution to the business problem where part of the solution is an IT application.
Our example illustrates this point. The top-level requirement is that the bank wants to sell within legal bounds because if it does not, the sale will damage the reputation of the bank and the bank may incur a fine. That single statement is the nub of the requirements, although you might need to fancy it up a bit with cost criteria and so forth. It is typical that the essence of a real high-level requirement in business is a single simple statement such as this one. Stating what a business wants to do is rarely complex; it’s figuring out how to do it that is. In our example, the response to this requirement is to design a business solution that consists of adapting a number of existing sales processes and adding some additional management processes (to monitor the answers) and marketing processes (to set the questions and analyze data gathered). In support of the business solution and integral to it, a new IT application is designed.
You may be thinking that design is too pompous a word for fixing a business problem. I suspect that this is partly because many business managers like a more free-flowing approach—trying something out and adapting it as they go along. Let me make it clear that a free-flowing exploratory approach is a perfectly acceptable way of doing design in general—this is discussed in more detail in the section called “Ad Hoc Design”—but it does not sit well with IT applications, at least with large applications. IT applications are rigid, brittle things, and while in this book I discuss how to make them more flexible and more responsive to change, this is much easier to do when you have a good understanding of which parts are likely to change. Put another way, you can make IT applications bendy but only in part. I once heard a certain well-known application software product described as being “as flexible as concrete”; it can be molded into any shape you like when wet, but when it sets, that’s it. A lot of IT applications are like that.
Let us see how turning “gathering requirements” into “designing a business solution” makes our assumptions come true. The key stage in any design is what I call the “design hypothesis”—it is the basic idea for the solution. In our example the design hypothesis is that during a sale, questions are asked and the answers are recorded. The most time-consuming part of the design process is elaborating the design, by which I mean working out what it means in practice and tidying up all the details. The big difference between a design approach and a “requirements gathering” approach is that in a design approach you spend little time asking people to fill in a blank sheet of paper and a lot of time presenting them with a possible solution and listening to their feedback. People find it much easier to criticize than to create. If asked for their needs, they will have difficulty listing them all; but presented with one or two possible solutions, they will quickly identify additional needs, errors, issues, and problems. Furthermore, differences of opinion among the stakeholders become apparent quickly when they are all reviewing the same design. The design provides a framework from which you can try to find a compromise. Here are our assumptions:
- Assumption one: clarity. You are not relying on the clarity of the people giving you requirements; you are relying on your own clarity to present to them an accurate picture of the solution.
- Assumption two: no disagreements. It is easier for people to raise alternative approaches when they can frame them as criticisms of a possible solution because it allows stakeholders to express disagreement without directly criticizing their peers. If disagreements are in the open, they can be more easily resolved.
- Assumption three: no unknown or ignored stakeholders. With the emphasis on feedback and openness it is easier to include people simply by sending them the electronic copy of the design presentation.
- Assumption four: feedback is clear. It is easier for the stakeholders to describe precisely how the solution is wrong than to describe what the solution should look like.
- Assumption five: cost estimates and trade-offs are included in the design process. Actually, I don’t think you can give a cost estimate from a business design alone; you have to go one step further and design the technical solution before making an accurate cost estimate. This is discussed in more detail in Chapter 3, “Reusing Existing Methods and Practices.” The good news is that taking this further step is not time-consuming in relation to the total time of the project. It is sufficiently short that it is reasonable to design alternative business solutions and alternative technical designs (for instance, with different availability goals) so that the organization can make an informed decision that suits its time frame and the size of its wallet.
- Assumption six: business managers understand integration options sufficiently that they can provide guidance in this area. As we shall see, integration options are presented as part of the design, not in techno-speak but in terms of access to data and passing messages.
The output from designing requirements is what I call a context design because it provides the context for the IT design. As you can see from my previous comments, it is vitally important that the context design give a clear picture to business managers of what the proposed IT application does.
I call the whole design approach context-driven design.
But before I discuss the other two points I raised at the beginning of this chapter (being more like engineering and working for rather than against the architecture), I need to discuss design in general.