- Success and Failure of Projects and Strategies
- Core Competencies
- The Need for Understanding: Abstraction, Precision, Explicitness
- Abstraction: The Way to Put Management in Control
- Basic Structuring Constructs
- Business Rules: Precision vs. Handwaving
- Tacit Assumptions and "Evident Truths"
- Specifying Problems and Solutions
- Where to Start and Why: Business Domains
Abstraction: The Way to Put Management in Control
Abstraction (suppression of irrelevant detail) is essential for any kind of human understanding and therefore for decision making. In the specific context of large IT system design and development, J.I. Schwartz noted in 1969 that "control is needed by managers, so that they don't begin a task before the initial specification is clear" [SE1970]. Clarity may be obtained only by using abstraction.
As C.A.R. Hoare observed at the same conference, if for some task you don't know what you have to do, "you can start out with the hypothesis that it will take a great number of people. Following down this slippery slope of reasoning, as soon as you have a very large number of people you have a very large management problem" [SE1970]. In this situation, management is not and probably cannot be in control; i.e., management cannot make justifiable decisions with the objective to improve the well-being of the owners of the enterprise.
Abstraction leads to elegant and very useful insights in all areas of human endeavor. When the irrelevant details have been eliminated and only the essentials remain, the number of potential blind alleys is drastically reduced. (For example, the essential properties of a thing cannot be changed without changing the thing.) At the same time, commonalities between seemingly different areas (often using quite different languages) may become apparent. Generalizations based on the essential properties of specific situations become formulated. The "Aha!" is being distilled.
Abstraction is indispensable for establishing that rapport between individuals that is "a very significant factor in arriving at agreed upon specifications from which one can proceed to build a programming system... It is quite evident you cannot have an effective software engineering facility when you lack rapport" (J.D. Aron, [SE1970], in the context of the Apollo project). This observation is valid not only in the context of programming systems. The rapport between people may be achieved only if some characteristics of people and things are by mutual (implicit or explicit) agreement considered to be irrelevant10 and thus ignored; the remaining characteristics will provide for a fruitful communication between the participants. Indeed, "[w]e get acquainted with another culture in the same manner as with another person: at our first meeting we look for commonalities in order for an acquaintance to become possible, and later we look for differences in order for the acquaintance to become interesting" (Averintsev's metaphor as described in [G2000]). It is instructive to observe that what was ignored as accidental when the communication was established becomes essential for the communication to become and remain interesting. At the same time, the essentials that made the communication possible remain as the fundamental context, when the communication participants concentrate on whatever at a higher abstraction level was considered to be accidental. For an important example of communication difficulties in the programming environment, of substantial interest to management at various levels, consider the infamous buffer overflows that have led to expensive and even dangerous consequences for too many IT users. A plausible explanation of such blindingly obvious violations of good programming practice (corresponding to violations of the most basic safeguards found in professional engineering) was presented in [N2002]: the programming culture of disciplined programming emphasizing the correctness concern differs from the programming culture emphasizing the efficiency concern in which it is imperative for "your dancing pigs [not to] dance slower than the other guy's."
The commonalities between cultures describe the essential structure of the subject matter of the cultures (for example, businesses). And the differences describe the specific nature of each culture.11 Following the discoveries made in present-day mathematics,12 we note that the structures of the essential fragments of apparently very different domains are often the same because the relationships between the individuals specified in these domains are of the same kind. We describe and use these relationships throughout the book.
Codifying corporate strategy
"In the mathematics I can report no deficience, except that it be that men do not sufficiently understand the excellent use of the pure mathematics, in that they do remedy and cure many defects in the wit and faculties intellectual. For if the wit be too dull, they sharpen it; if too wandering, they fix it; if too inherent in the sense, they abstract it."
Roger Bacon (1214?-1294?)
In any domain, when we make important decisions we want to understand precisely what exists, where it is, and what it does [RM-ODP2]. A business specification of the domain defines exactly that.
In business modeling, we need to be able to discover and make explicit the structures13 inherent in the business but often hidden within a huge amount of detail. In addition, we need to be able to discover how reasoning about these structures may lead to improving various properties of the business. And, of course, we need to be able to define precisely the corporate strategy, i.e., what properties are of interest and what "improving" means! In order to do that, we need to be able to reuse as much as we can, starting with well-defined fundamental constructs. Definitions of these constructs often come from mathematics (or philosophy) and can be translated from mathematical terms into terms more familiar to the stakeholders. Although the elucidation and explicit specification of concepts and structures described here may have happened relatively recently, the concepts and structures themselves have been around for centuries and perhaps millennia, and have been mentioned with varying degrees of explicitness and precision for quite a while.
In business modeling, when we use, discover, and formulate business laws and business rules, we can follow the founders of economics. And in economics (a science of complex phenomena, according to Hayek), specifications and discussions of the essential structures may be found in classical works by Adam Smith and Francis Hutcheson, and in more recent works by von Mises, Hayek, and others. All these structures are abstractions; that is, they are obtained by suppressing irrelevant details. Specification of corporate business structures within their environment is essential for reasoning about corporate strategy and tactics.
With respect to logical reasoning (which goes back to Aristotle), we note the remark by von Mises [M1949]: "Logical thinking and real life are not two separate orbits. Logic is for man the only means to master the problems of reality." Alternatives to logical reasoningsuch as handwaving, reliance on those who "get it," or on business plans described by means of flashy presentations using box-and-line diagramsoften leads to business failures exemplified by many "dot-coms". As noted by Paul Strassman [S2001c], knowledge about a company's business environment and its competitive position is the most valuable insight for strategic decision making by executive management; 65 percent of the profit performance of corporations can be attributed to this knowledge.
As noted above, modeling is an area in which mathematical maturity is of great importance. Mathematical maturity is not the same as specific mathematical knowledge because mathematics in general is "the art and science of effective14 reasoning." In other words, mathematics promotes simplification rather than sophistication; by using abstraction, mathematics elucidates the essentials instead of creating a language barrier. Mathematics emphasizes the structurei.e., the relations among various objectsto a much greater extent than the properties of the objects themselves. As a result, we speak about the fundamental unity of all mathematics. In understanding and modeling a business, as in mathematics, we emphasize the structurei.e., the relations among various business things15to a much greater extent than the properties of the specific business things themselves. And as in mathematics, we can see a fundamental unity of the same kind in the basic concepts of all kinds of specifications. In this manner, the writers and readers of business, technological infrastructure, and IT system specifications use the same basic concepts and constructs and are able to communicate in precise and explicit terms.
Abstraction makes things simpler and makes understanding much easier. Concepts are understood much better than technicalities, and important results are essentially simple. Modeling experience suggests that business stakeholders (non-experts in modeling or mathematics) understand and use conceptual material very well.
As a result, we may speak about the profound commonalities among apparently different kinds of business. More specifically, we may speak about the commonalities between the "old" and the "new" economy or about discovering and reusing business laws (structural and behavioral) common to all businesses. In this manner, by using abstractionsuppression of irrelevant detailswe break through the traditional and not too helpful partitions separating different kinds of business.
The structuring of any system (business, IT, and so on) is where relationships, especially a small number of generic relationships encountered in all business, and other, specifications, come into play. The three kinds of generic relationships encountered everywhere and widely used in this book are composition, subtyping, and reference.
Concepts, like concrete things, do not exist in isolation: only a system of interrelated concepts makes it possible to be precise and explicit when the corporate strategy is discussed. We can bridge the communications gap between different people and departments when everyone can use the same well-definedand small!conceptual framework. And thus it becomes possible to make decisions based on reasoning about explicitly presented facts and structures within that framework rather than based on handwaving and eloquence. The fallacy of the latter kind of decision making was clearly visible in the failure of some "dot-coms" created, supported, and even valuated by those who "got it" as opposed to those who "didn't get it" [W2000].
Creating and maintaining elegant systems of concepts requires a considerable effort. Bypassing this effort is "about as appropriate for the software engineer ... as it is for the civil engineer to try to avoid expensive heavy earth-moving equipment, relying instead upon armies of men with baskets" (R.M. Needham, [SE1970]). Unfortunately, some important aspects of the current state of software engineering, especially the ones associated with "engineering" as a profession, can still be characterized by this quote. This is unacceptable. Producing and using demonstrably correct specifications as a foundation for making strategic decisions leads to success both in traditional businesses and in software engineering with further, and easier (due to reuse), success in traditional businesses.
Prerequisites for Decision Making
When we want to make reasonable decisions and justify these decisions to ourselves, our partners, customers, and other stakeholders, we need to base our decisions on the essentials of the information we have or can obtain while ignoring the accidentals. Choosing what is essential is often viewpoint-dependent. In other words, we need to apply abstraction in order to concentrate on the essentials, and precision in order to demonstrably "know what we are talking about." Thus, we act as scientists developing a theory of the business we are dealing with and validating this theory with the business stakeholders [CHJ1986].
These approaches have been around in traditional business and engineering for a long time. There has not been and cannot be any step-by-step guide showing how to create elegant specifications and how to use them in decision making. At the same time, important general guidelines exist.
Consider the Laws of the Problem Domain
The helmsman used to stand by with tears in his eyes; he knew it was all wrong, but alas! Rule 42 of the Code, "No one shall speak to the Man at the Helm," had been completed by the Bellman himself with the words "and the Man at the Helm shall speak to no one." So remonstrance was impossible, and no steering could be done till the next varnishing day. During these bewildering intervals the ship usually sailed backwards.
Lewis Carroll, The Hunting of the Snark [C1876]
Before solving a problem, we need to be able to understand what the problem is about. This approach is used in other areas of human endeavor, but not always in information management. Many of us have heard about solutions in search of problems. And many have heard about universal solutions apparently applicable to any problem of somenot always well-definedtype.
Before trying to solve a business problem by means of an IT solution, we need to understand the business context (i.e., the business laws, as well as explicit and implicit rules) and the specifics of the particular problem. Usually, this is the most difficult aspect of the job. The business context defines the properties of the business (things, relationships, and actions16) in the same manner as laws of nature define the properties of nature. In the context of IT system development, the need for "concept formulation at the beginning" was explicitly mentioned at least as early as 1969 by J.D. Aron from IBM Federal Systems Division [SE1970]. In order to be understood, the business problem should be explicitly formulated in terms of these essential properties of the business.
Understanding of a business problem may be compared with a "business diagnosis" (ICSE 2001 talk by J. Ning from Accenture). When we use this metaphor, we may recall that in medicine the normal characteristics of a human bodyboth static and dynamicare supposed to be well known before dealing with its "abnormal" characteristics. This includes the determination of "normality" of the characteristics.
After we understand the business and the specific problem in the context of that business, we may see that a computer-based solution is not even needed: manual changes in business processes may help the business stakeholders to understand better the current business objectives and to achieve them. At the same time, in many situations, IT (computing support) may help the business to achieve its existing objectives or may provide for new business opportunities that may become or change, after approval by business decision makers, the current business objectives. In either case, the IT artefacts must be understood by the business decision makers and specified from the business viewpoint. This approach does notand should notdiffer from applying any technology in a business. For example, if we want to "get from here to there" we need to know some aspects of the map (which will show the relationships between Here and There17) and some aspects of the technologythe transportation mechanisms available to us.
Understanding and precise specification of the (laws of the) appropriate domains18 is the essential prerequisite for understanding both the business and the IT artefacts,19 and in particular, for trying to formulate any requirements for computing support. In other words, both the business problem and its solution could be understood only in the context of their business and solution domains since the laws of these domains ought to be satisfied all the time. This is as it should be: in more established branches of engineering, for example, we consider the laws of nature (specified, e.g., in physics) as a natural conceptual framework for dealing with engineering problems and artefacts. In software engineering, we ought to apply the same approach.
Of course, there are differences between traditional and software engineering. Because laws underlying software engineering are relatively new, they may be less familiar than those underlying traditional engineering; some of these new laws have not been formulated yet. And because software engineering is applicable in many very different business areas, we ought to be able to find out how to specify in a uniform manner the laws of these business areas. Fortunately, mathematics, being the art and science of effective reasoning, helps us to do just that. Thus, we will see that specifications of very different businesses areas (including the businesses of software engineering) have much in common.
Some models of various application domains and their fragments are (publicly or otherwise) available, so in software engineeringof which business modeling is an essential partit is often necessary to create new models or at least to evaluate and change models created by others, rather than merely to apply such models.
Requirements "Always Change" ... or Do They?
Business processes often change: such changes may lead to a competitive advantage for the business, or may simply be perceived as necessary for the business to survive. Similarly, decisions about using computer-based IT systems to automate certain business processes may also change (for example, due to perceived opportunities). At the same time, the basics of a business domain often remain the same for centuries if not for millennia: we successfully used banking and financial texts published in the early twentieth century (or earlierfor example, fragments from Adam Smith's Wealth of Nations) in order to understand and specify the corresponding business domains. The changes due to "modernity" were minimal and were mostly additions or refinements of the existing classical models.
It is interesting to note that in literature and art, modernity is an attempt to derive the eternal from the transitory. In the words of Baudelaire, "Modernity is the transitory, the fleeting, the contingent, one half of Art, whose other half is the eternal and immutable" [P2001a]. In our terminology, invariantsproperties of things and relationships that remain true no matter what processes have been or will be performedprovide for the description of this immutable. This terminology has also been used by some scholars of humanities. For example, linguistic invariants provided the foundation of many deep discoveries by Roman Jakobson. For another example, the interesting book by Zholkovsky [Z1999] is about the discovery of a small number of invariants in the life and literary work of Zoshchenko; these invariants help in understanding the seemingly very different kinds of Zoshchenko's creative work.
The invariants that define the business domain are a very good and stable foundation for a specification.20 We do not want to start in the middle, that is, with a possible solution, or even with a specific problem; we start with the stable (that is, invariant) basics and proceed from there. In other words, we use abstraction to concentrate on the essential and ignore the transitory. The transitory cannot be understood without the basics. In particular, the interrelated concepts used in describing this transitory are defined by (and in) the basics.
The invariants are the "laws of the domain," and we use them in the same manner as laws of nature are used in more traditional engineering. And the basic laws of a domainunlike the transitory21are usually simple and elegant, as are the basic laws of nature. These laws of the domain constrain and govern all actions that happen in that domain; specifically, human participants of the domain use the invariants in planning their actions to improve their future conditions. In this manner, the stable structure of a business domain provides an excellent reusable foundation for creating and reasoning about all kinds of potentially volatilebut not frighteningly volatilerequirements for business processes and systems.
The invariants that define the basic structure of a business domain also may change. However, such changes happen much more seldom than changes in the transitory characteristics of the domain (sometimes called "business needs") or changes in behavior (business processes) governed by these invariants. In addition, business specifications based on the invariants have delivered an unusually high degree of robustness [KA1997]; almost all changes have been local. The structure of the business domain usually remains stable, while the changes of the contents often are about permission of prohibition of certain products or services, introducing of a product or service into new contexts, deregulation leading to competition where none previously existed, and so on. The general properties (but, of course, not the specifics) of many of these changes can often be anticipated, and appropriate abstraction leads to substantially greater stability of specifications.
Strive for Simplicity and Elegance
Association with ugliness blunts the instinct for beauty, and warps the intellect if long continued, whereas association with beauty sharpens instinct and allows the intellect to develop without hindrance. Beauty thus being hygienic, art should come into everything, and the architect, like every other worker, should make his work beautiful.
H. Kempton-Dyson, Design, in [S1904]
The stable foundations often ought to be discovered and distilled from the complex world around us. These discoveries are the only way to master the complexity and abstract out the irrelevant details.
Unmastered complexity can creep in various ways. "Too much stuff" is probably the most familiar22 and the most dangerous. A specification with too much unstructured stuff in it is a write-only specification, even if most names used in it are perceived to be very familiar to its possible users. Most of us have seen such specifications materialize in boxes full of precise information; if we want to ask a question, the answer is probably there, but impossible to find. Decision makers and others who have seen only such specifications may become convinced that all specifications are not worth their time and effort. Some practitioners may, as a result, strive for "quick and dirty" work with often disastrous consequences.
As E.W. Dijkstra noticed some time ago, "I get extremely suspicious when the engineer justifies adhoccery by an appeal to the presumed law of nature, summarized by 'quick and dirty,' for from my experience and from my understanding I feel that 'quick and elegant' is a much more likely combination" [D1969]. This adhoccery is often realized by following the appropriate buzzwords and starting from scratch without paying much (or any) attention to the basics of the discipline. Contrariwise, quick and elegant specifications help enormously in the collective work of business experts, modelers, and IT experts. (Most specifications shown in this book are generalized fragments of real-life business specifications.)
In business and IT system specifications, too much stuff is often realized as overuse of a huge number of detailed facts possibly including application-specific or technology-specific artefacts invented for the benefit of the initiated. Such a specification may be precise and even may be claimed to be complete, but who can read it? If we want to drive from New Jersey to California and we can use only very detailed road maps (e.g., having the scale of one mile to one inch) plus a "data dictionary" such as an alphabetized list of towns, rivers, and counties, we are in a very difficult situation; based on these complete and precise specifications, we cannot even figure out that we have to drive west! In addition, some specifications introduce new terminology invented for an apparently new area, while in fact many existing concepts could have been reused. We often see proponents of reuse who propose first of all to create a set of new and therefore better reusable constructs.
According to Ronald Posner [P2000], semiotic pollution harms our intellectual environment in the same manner as physical pollution harms our physical environment. Information overload often leads to our intellectual incapacitation: we are overwhelmed by complexity. "The most effective way of avoiding unmastered complexity is not to introduce that complexity in the first place" [D2000]. After such complexity has been introduced into a system, it is usually too late: only trivial changes can be "reliably" made to that system unless a complete rewrite is accomplished. Even changes considered to be trivial (i.e., apparently local) may lead to grave consequences if the locality assumption, often unstated, will appear to be mistaken. Some of these grave consequences were described as front page news. And similar problems have been well-known in traditional environmental pollution.
A system with unmastered complexity is far from elegant; "in the design of sophisticated digital systems, elegance is not a dispensable luxury but a matter of life and death, being a major factor that decides between success and failure" [D2000]. The same is true about any system. And Paul Erd?, one of the greatest mathematicians of the twentieth century, put forward the idea that the Creator has a Book in which all of the most elegant proofs are written [S2001].
In the same manner as many basic business concepts (e.g., those described in Adam Smith's Wealth of Nations or in the United States Uniform Commercial Code) are neutral with respect to a specific business, many basic IT concepts are neutral with respect to a specific technology (and of course, to a specific methodology or toolset). This approach has been used by E.F. Codd, the founding father of the simple and elegant relational data model, as opposed to navigational models that "burdened application programmers with numerous concepts that were irrelevant to their data retrieval and manipulation tasks, forcing them to think and code at a needlessly low level of structural detail" [C1982]. More generally, this approach has been used in excellent programming texts (such as [D1976, D1982]) that teach programming concepts rather than a specific programming language. It leads to simplicity and understandability, and it is not new. As an example, the concept of Occam's razor was formulated long ago and has been successfully used for centuries. The same approach has been successfully applied to humanities. Johann Joachim Winckelmann, an eighteenth-century German aesthetician, noted that it was easy to use lots of means to produce insignificant works while it was difficult to create real masterpiecesbeautiful works (of art) with simple means. These observations were made in a very interesting paper on Web design simplicity [K2000a].
Using mathematics helps in avoiding too much stuff for any system. Yuri Lotman, one of the founders of semiotics, noted that mathematics is "a method of scientific thought, and a methodological basis for the discovery of the most general regularities of life" [L1964]. Observe that Lotman considered the main difficulty in using mathematical methods in literary studies to be "in the fact that the basic concepts of literary science have not yet been formulated." As an IT-related example, Jim Davies and Jim Woodcock from the University of Oxford presented at the Oxford-Microsoft Symposium in Honour of Sir Tony Hoare (in 1999) an extraordinarily successful project of a smart card (for electronic commerce) for a very large U.K. bankthe first U.K. system to gain the Level 6 security rating that was considered unattainable by the industry. Formal specification, design and implementation (with proof) were successfully used in the project. The emphasis was on simplicity: it made the specification "blindingly obvious," so that the specifiers could convince a bank manager that the specification was correct. The presenters stressed that their most interesting examples were characterized by eureka moments. The mathematics required for the specification, refinement, and proof was completed on time and under budget, and there was no problem of "using Greek letters"!
Some authors (E.W. Dijkstra, J.W. Smith [SE1969]) advise to use not more than several lines (or a paragraph) in order to explain something specific, be it in mathematics or specifications. This approach perfectly corresponds to presenting the main characteristics of a language used to express a specification on the back of the proverbial envelope. We try to follow these recommendations throughout the book.
Simplicity should never be at the expense of generality. Discovering commonalities in areas perceived to be somewhat or substantially different leads to generalization and simplification (by virtue of suppressing irrelevant details), and thus to better understanding. As a result, effective recurring templates may be discovered or distilled and specified for reuse. The simple, general, and powerful concept of a contract described, for example, by Francis Hutcheson [H1755] and by the U.S. Uniform Commercial Codeas a composite in the composition of parties, subject matter, and considerationrepresents a well-known example used in a variety of contexts.
We have to communicate our specifications, i.e., to represent them in a language or languages understandable to the users of these specifications.23 There are many specification languages, in the same manner as there are many programming languages. A programming language may be ugly, but at the very least it has to be precise because it includes instructions to an inanimate object (a computing system). A specification language may be ugly and also (independently) may be imprecise because it may be used only by humans. Box-and-line diagrams,24 slide shows, narratives, and so on are familiar examples. This book does not promote a "best" specification language, but rather describes some criteria that make a language good.
A good specification language should first be simple and elegant. It should not add its own complexity to the intrinsic complexity of the problem. "All knowledge of one's programming language can conveniently be formulated on a single sheet of paper and can be gotten across the limelight in about twenty minutes lecturing.... [O]nly then real difficulties of understanding and solving real problems can be dealt with; that activity requires the ability to think effectively more than anything else" [D1976a]. Thus, again we encounter the need for a well-defined elegant structure. The same approach applies to languages used for non-executable programs, i.e., for specifications considered in this book.
A good specification language should be precise. In other words, the semantics of its concepts and constructs and the relationships between them should be explicitly and unambiguously defined. Thus, if diagrams are used, then they should not be pictures of anything; rather, all elements of these diagrams should represent mathematical objects. (As a counter-example, consider a line between two boxes that intends to represent a relationship graphically; the semantics of such a representation is not clear; different lines that look the same often denote different semantics; and in most cases, a relationship has more than two participants.) Of course, we do not need to repeat the definition of a construct every time we use it; once defined, the construct may be abbreviated (named) and reused. We reuse such constructs, from very basic to very specific, in this book.
A good specification language should permit the specifier to say what is meant without any language-imposed restrictions: it should permit an easy mapping from the structure of the problem to the structure of its representation. This is a very important characteristic of any notation: to quote E.W. Dijkstra, "the tools we are trying to use and the language or notation we are using to express or record our thoughts are the major factors determining what we can think or express at all" [D1972]. The specifiers should be able to, and should be encouraged to, say what they mean rather than something quite different.25 For example, if the specifier means to say that there exist several independent subtyping hierarchies for the same supertype, then the specifier should say precisely this, and the specification language should be able to express this directly and explicitly, rather than prohibiting such an expression "because you cannot do that in [insert your favorite programming language]." For another example, if the specifier means to say that a composite in a composition has three components of different types, then the specifier should say precisely that, and the specification language should be able to express that directly and explicitly, rather than prohibiting such an expression and imposing instead three binary compositions that collectively have a very different semantics.