Home > Articles > Programming

This chapter is from the book

1.3 Electronic Archeology: Layers upon Layers

Enterprise systems are distributed, event-driven systems. But they all have another property in common. They are layered systems. Layering is a design technique for controlling complexity. But it is also a side e'ect of unplanned or uncontrolled growth. In some systems the situation bears analogy with the multiple layers of civilizations of di'erent historical periods, one on top of another, that the archeologist Heinrich Schliemann discovered at Troy, back in the 1870s.

Layered IT systems present another dimension in the search for new ways to understand the events that happen in them.

1.3.1 A Layered Enterprise System

Figure 1.4 shows a common layered structure of an enterprise system.

Figure 1.4Figure 1.4 Typical layers in an enterprise system

Applications and Users at the Top

The top layer is called the business level or strategic planning level because it is the layer at which an enterprise plans and transacts its business. It contains the applications that people use to transact business and do their work. This layer contains user interfaces to the enterprise's services, planning and forecasting applications, inventory and accounting applications, spreadsheets, calendars, Web browsers, document preparation tools, e-mail, communication devices such as cell phones, system administration tools, and so on. The contents of this layer are expanding rapidly. Di'erent kinds of applications are appearing all the time, each requiring di'erent types of information.

The application layer is the level at which human users work and conceptualize their goals—for example, "I want to send e-mail to so-and-so," or "I want to initiate a process to purchase item X from the cheapest supplier." Events that signify activities at this level are the ones humans can understand most easily. These events result from using applications.

The high-level events that are of greatest significance are not explicitly generated by users using applications, but they are consequences of sets of other high-level events. For example, a single business-level event could consist of a sequence of events indicating uses of various applications. It could signify a completed supply chain transaction with business partners, negotiated by messages across the Internet. It is a virtual event signifying a very real activity with contractual and economic implications. These inferred events are aggregated from sets of other high-level events and are the events of interest to the business level or strategic planning level of the enterprise.

Another example of a complex business level event is a stockbroker, or a group of brokers, in a financial trading enterprise trading stocks on many di'erent accounts, including those owned by the trading house itself. Some patterns of trading activities on various accounts might violate Securities and Exchange Commission (SEC) regulations. But high-level events signifying trading violations have to be inferred from the explicit trade events generated by the brokers when they make trades. It depends on an ability to recognize complex patterns of events involving timing and relationships between the events. We understand a "violation event" when it is pointed out to us—no problem! But it isn't actually generated. That's the problem! It is a virtual event signifying a very real activity. It has to be recognized from the morass of actual events.

There are myriads of similar situations going on in enterprise systems all the time that have the potential for similar kinds of violations of the enterprise's policies—for example, in online Internet-based auction houses and in "dot-com" retailing, where violations of the regulations of various states may happen.

Conceptually, the events of interest to the highest layer of an enterprise, or a system of multiple cooperating enterprises, are virtual events comprising sets of application layer events. The strategic planning layer must be fed aggregations of application-level events.

But application-level events are not subjected to a battery of monitoring and analysis tools as the network-level events below it are. This is unfortunate but understandable, because such tools would need a new kind of technology that deals with complex patterns of events to be e'ective.

So, in a layered system, understanding at the highest level what is going on in the enterprise, at every layer, is one problem. But there are more problems to come because causality between events crosses layers.

Collaboration and Enabling Layer

Next in this figure comes the collaboration layer, which contains components that help make applications available to users. Here we find components that we have come to expect, such as databases and servers for e-mail groups, news, and chat rooms. We might even go so far as to include the operating systems, such as Windows and Linux, in this layer, although it could be argued that they belong in layers below. The collaboration layer now contains a growing number of products that enable more sophisticated uses of distributed information processing. For example, Web sites that service applications (so-called application service providers, or ASPs). And business rules engines that enable trading agreements and correlation of information. The collaboration layer contains more and more tools that enable users and applications to interact intelligently on complex projects, like building software systems or putting together multistep trading deals. This layer is the target of a growing industry in business-to-business products and services (often called B2B).

The dividing line between the collaboration layer and the application layer is not a definite, rigorous line. Sometimes, for example, a database would be more appropriately viewed as an application rather than as an enabler of applications, in which case we would put it in the layer above.

Middleware

The next layer down is the middleware layer, which lets all the stu' above communicate. Included are Object Request Brokers (ORBs) and messaging systems such as information buses. This level of communications sits on top of the basic networks and lets all the applications and application servers talk to one another. It contains the system components that are often categorized as Enterprise Application Integration (EAI) products. It is called "integration" because it helps link business-level applications by letting them communicate. And it is called the middleware layer because its components are viewed as lying "in the middle" between applications and networks.

Before 1990 this layer and the products in it didn't exist. It came about by design, to make it easier to get applications to communicate over networks. Middleware is a go-between, translating the message communication between applications into low-level messages called packets that networks are designed to carry.

Communications middleware has become an industry in itself. Middle-ware for distributed applications includes ORBs and various kinds of message-oriented middleware based upon message queues, publish/subscribe (pub/sub) services, and so on. These products provide a layer of communication protocols, together with Application Program Interfaces (APIs), that the enterprise business applications use to communicate. At present there are several widely used commercial middleware products that form the basis for the EAI industry.

The Under-the-Hood Layer

At the bottom is the network layer—the essential plumbing for transporting information from one point to another, both within an enterprise and across enterprises. The kind of networks we are talking about here first appeared in the 1970s. Before that, applications had no way to communicate with one another.

The network layer is the core of the information technology layer (IT layer) in an enterprise system. Newspaper articles refer to it as the "under-the-hood" part of an enterprise's IT system, or the Internet for that matter. It is generally looked upon as something the common man should not know about and certainly not tinker with—it is a source of many system problems. And when it collapses in one of many well-known or not so well-known ways, the whole system grinds to a halt. We often hear "The network is down." Network crashes can become a critical concern to the higher-level echelons in a distributed enterprise. And they certainly a'ect business. Spectacular network crashes make newspaper headlines—for example, when online stock-trading systems clog up and stop the customers from trading, or when hackers orchestrate a denial-of-service attack on Yahoo and other popular Web sites.

So the network layer has become the domain of a powerful new kind of expert, the specialist in network and IT systems management. Often this person has a title, like IT director. The job is to keep the network layer running at all costs—otherwise, the enterprise is out of business. It's the first point of pain for a distributed enterprise.

Consequently, network management products have proliferated. There are all sorts of tools for the IT director to use. They track resource consumption (CPU use and memory or disk space use) on every machine in the network and provide statistics and summaries in every color in the rainbow. They log alerts or warnings from network components such as routers and servers, slice and dice them by the minute or hour, raise the alarm, and page the IT director out of bed at any hour of the day or night.

But keeping the network running is not the IT director's only problem. Additionally, there are issues of security and privacy. And cyber warfare is an increasing headache for all enterprises. Everyone knows it is going to increase. Plenty of tools and products try to help the IT director solve cyber warfare problems, but again, they don't do too well.

The IT director's problem is making sense of all this network-level information. The importance of this task is well understood by enterprises today. They invest heavily in whatever technology is available to help. For example, the network operations room in a large global banking enterprise represents a billion-dollar investment these days.

1.3.2 Vertical Causality: Tracking Events up and down the Layers

Although there are no general layering standards for categorizing enterprise systems, the picture in Figure 1.4 is typical and captures the general idea.

  • The applications that humans use and understand (sometimes) are at the top.

  • Network plumbing that carries the messages and information is at the bottom.

  • Several layers are in between, depending upon the complexity of the system.

Activity at each layer is translated into activities at the layers below and conversely. Those lower-level activities must complete successfully in order for the higher-level activities to also complete successfully.

So, we have a general principle of layered enterprise systems: An activity at the top causes activities at successively lower levels, which in turn cause other activities to happen at the top.

Layering adds another dimension to understanding events. Discovering the causal relationships across layers, between high- and low-level activities, especially in real time, is another outstanding issue in enterprise systems. We call this vertical causality.

An ability to track vertical causality in a layered system has at least two important uses in managing our enterprises. First, knowing how a business-level event is broken down by your system into sets of lower-level events is very helpful in understanding properties of that high-level event, such as its timing—for example, why it took so long or failed to complete—and how to improve performance at the business level. Sometimes we may only need to understand the breakdown of the business-level event into calls to applications, while at other times, we may need a breakdown all the way to the network level. This is the kind of information that might have helped the IT manager answer the CEO's question about what caused customers in the New York area to complain about slow service.

Second, if we were able to track vertical causality, we could use it to group events at lower levels according to the high-level events they signify. That would help us make sense out of the mountains of low-level events that network monitoring tools give us.

Tracking vertical causality is not a simple problem because there is almost as much dynamism in the layers of our enterprise systems as there is in the communication between them across the Internet. For example, the ability of a service provider to load-balance incoming requests may depend on the loads of lower-level servers that are continually changing. A high-level complex event, such as defaulting on a service-level agreement, can happen in many di'erent ways. It may result from lots of di'erent sets of lower-level events. Moreover, our enterprise systems are dynamic in their composition and structure. The sets of components at each level are changing all the time, as are the kinds of activities the system is being employed to do. So the kinds of events that can happen at every level are changing too. Keeping up with what our systems do is not an easy task.

1.3.3 Event Aggregation: Making High-Level Sense out of Low-Level Events

What about those strategic-level complex virtual events that we talked about earlier? Virtual events are not actually generated in the system—nobody sends a message saying, "I violated law XYZ." The activities sig-nified by high-level virtual events have to be recognized from among sets of events at the application layer or below. These policy and contractual events are top-level events of immediate interest to the corporate level of the enterprise.

We have just discussed the potential usefulness of an ability to track vertical causality. But there needs to be a high-level event in order to track its causes. Often, all there is at the highest levels is a perception, a suspicion, a list of complaints. But the business and strategic implications are very real.

The complementary operation to downward tracking of vertical causality is the aggregation of sets or groups of lower-level events into a single higher-level event that expresses the meaning of the lower-level events, taken together. These are some examples:

  • A group of stock-trading events, related by accounts, timing and other data, taken together constitute a violation of a policy or regulation.

  • A large set of network-level operations, originating from the same machine and repeating similar accesses on di'erent target machines, may signify an attempt to intrude into a network.

  • In an application service provider, groups of events signifying accesses to distributed applications, such as those illustrated in Figure 1.3, together may imply a violation of a service-level agreement with a customer.

Recognizing or detecting a significant group of lower-level events from among all the enterprise event tra=c, and creating a single event that summarizes in its data their significance, is called event aggregation. The aggregate event will appear at a higher level in the enterprise's operations.

Event aggregation is a capability that needs to be developed. It is a very powerful technology for monitoring and managing our enterprises. It will in turn depend upon technology for recognizing patterns of events in large amounts of lower-level event tra=c, in real time. And it depends first on an ability to express patterns consisting of multiple events together with their common data and timing. If event aggregation is implemented properly, it can give us the ability to track the lower-level events that were aggregated to create a high-level event—so-called drill-down diagnostics for tracking vertical causality.

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.

Overview


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information


To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information


Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children


This site is not directed to children under the age of 13.

Marketing


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information


If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information


Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents


California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure


Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact


Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice


We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020