Home > Store

Power of Events, The: An Introduction to Complex Event Processing in Distributed Enterprise Systems

Register your product to gain access to bonus material or receive a coupon.

Power of Events, The: An Introduction to Complex Event Processing in Distributed Enterprise Systems

Premium Website

  • Sorry, this book is no longer in print.
Not for Sale

Description

  • Copyright 2002
  • Edition: 1st
  • Premium Website
  • ISBN-10: 0-201-72789-7
  • ISBN-13: 978-0-201-72789-0

As global information systems become ever more powerful, complex, and intertwined, companies need new approaches to extracting information, transforming it into intelligence, and acting on it. David Luckham introduces a breakthrough solution that offers compelling benefits at every level and scale of the enterprise: Complex Event Processing (CEP). Luckham first identifies key challenges faced by today's enterprise information systems, and demonstrates the "event-driven" nature of management in the electronic enterprise. He then introduces CEP, showing how it can harness the power of events to automate management without compromising managers' control. Luckham illuminates fundamental concepts such as events, causality, event hierarchies, event patterns, and rules; then shows how these concepts can be used to solve key enterprise management problems. In Part III, he presents a realistic description of what it takes to build CEP applications that scale to real world problems, introducing the new RAPIDE event pattern language. The book concludes with detailed case studies that show Luckham's CEP tools at work in the enterprise.

Sample Content

Online Sample Chapters

The Global Information Society and the Need for New Technology

The RAPIDE Event Pattern Language

Downloadable Sample Chapter

Click below for Sample Chapter(s) related to this title:
Sample Chapter 1
Sample Chapter 8

Table of Contents



Preface.


Acknowledgments.

I. A SIMPLE INTRODUCTION TO COMPLEX EVENT PROCESSING.

1. The Global Information Society and the Need for New Technology.

Distributed Information Systems Everywhere.

The Global Communication Spaghetti Pot.

Event Causality.

Electronic Archeology: Layers upon Layers.

A Layered Enterprise System.

Vertical Causality: Tracking Events up and down the Layers.

Event Aggregation: Making High-Level Sense out of Low-Level Events.

The Gathering Storm of New Activities on the Web.

Global Electronic Trade.

Agile Systems.

Cyber Warfare and the Open Electronic Society.

Summary: Staying ahead of Chaos.

2. Managing the Electronic Enterprise in the Global Event Cloud.

How the Global Event Cloud Forms.

The Open Enterprise.

The Global Event Cloud.

The Electronic Enterprise.

Operating in the Global Event Cloud.

Going Beyond Workflow.

Parallel and Asynchronous Processes.

On-the-Fly Process Evolution.

Exceptions Must Be First-Class Citizens in Process Design.

Summary: Managing the Electronic Enterprise.

3. Viewing the Electronic Enterprise-Keeping the Human in Control.

Today's Event Monitoring Is Too Primitive.

System Monitoring Focuses on the Network Layer.

Network-Level Monitoring Doesn't Even Solve Network Problems.

An Example of Causal Tracking.

Information Gaps.

Examples of Information Gaps.

Problem-Relevant Information.

Viewing Enterprise Systems.

Creating and Coordinating Multiple Views.

Hierarchical Viewing.

An Example of Hierarchical Viewing.

Summary: Viewing the Electronic Enterprise.

4. Designing the Electronic Enterprise.

Process Architectures.

Roles of Architecture in the Process Lifecycle.

Constituents of Process Architectures.

Annotations.

Architectural Structure.

Interface Communication Architectures.

Architecture Diagrams.

Behavior Specification.

Design Constraints.

Examples of Informal Annotations.

Dynamic Process Architectures.

Diagrams for Dynamic Architectures?

Layered Architectures and Plug-and-Play.

Abstraction Principle.

Summary: Technology to Support Process Architecture.

5. Events, Timing, and Causality.

What Events Are.

How Events Are Created.

Time, Causality, and Aggregation.

The Cause-Time Axiom.

Genetic Parameters in Events.

Timestamps.

Causal Vectors.

Time.

Causality and Posets.

Causal Event Executions-Real-Time Posets.

Orderly Observation.

Observation and Uncertainty.

Summary.

6. Event Patterns, Rules, and Constraints.

Common Kinds of Pattern Searching.

Event Patterns.

A Strawman Pattern Language.

Pattern Matching.

Writing Patterns in STRAW-EPL.

Event Pattern Rules.

Constraints.

Summary.

7. Complex Events and Event Hierarchies.

Aggregation and Complex Events.

Creating Complex Events.

Event Abstraction Hierarchies.

Viewing a Fabrication Line.

Building Personalized Concept Abstraction Hierarchies.

Viewing Network Activity.

Viewing Stock-Trading Activity.

Summary.

II. BUILDING SOLUTIONS WITH CEP.

8. The RAPIDE Pattern Language.

Event Pattern Languages-Basic Requirements.

Features of Rapide.

Types.

Predefined Types.

Structured Types.

Event Types.

Execution Types.

Subtyping of Executions.

Attributes of Events.

Basic Event Patterns.

Placeholders and Pattern Matching.

Matching Basic Event Patterns.

Placeholder Bindings.

Notation to Aid in Writing Patterns.

Relational Operators and Complex Patterns.

Relational Operators.

Guarded Patterns.

Content-Based Pattern Matching.

Context-Based Pattern Matching.

Temporal Operators.

Repetitive Patterns.

Pattern Macros.

Summary.

9. CEP Rules and Agents.

Overview.

Event Pattern Rules.

Definition of Event Pattern Rules.

Rule Bodies.

Context and Visibility Laws.

Semantics of Event Pattern Rules.

Examples of Rules.

Event Processing Agents.

Definition of EPAs.

Semantics of EPAs.

Event Pattern Filters.

Definition of Filters.

Semantics of Filters.

Action Name Filters.

Content Filters

Context Filters.

Event Pattern Maps.

Definition of Maps.

Semantics of Maps.

Event Pattern Constraints.

Definition of Constraints.

Semantics of Constraints.

Examples of Constraints.

Other Classes of EPAs.

Summary.

10. Event Processing Networks.

Common Structures of EPNs.

Flexibility of Event Processing Networks.

Connecting Event Processing Agents.

Basic Connections.

Guarded Connections.

Multiple Basic Connections.

Dynamic Event Processing Networks.

Class Connections.

Creation and Termination Rules.

Connection Generators.

Architectures and Event Processing Networks.

Architecture Classes.

Semantics of Architecture Classes.

Examples of EPNs and Architectures.

Case Study: EPNs for Network Viewing.

Visual Tools for Constructing EPNs.

Security.

Scalability.

Summary.

11. Causal Models and Causal Maps.

Causality between Events, Revisited.

Why We Need Causal Models.

What Causal Models Are.

Defining a Causal Model and a Causal Map.

Using Pattern Pairs to Specify Causal Models.

Using Causal Rules.

Resolving Ambiguities.

Causal Maps.

A Small Example of a Causal Map.

A Second Example of a Causal Map.

Developing Accurate Causal Models.

Summary.

12. Case Study: Viewing Collaboration between Business Processes.

A Collaborative Business Agreement.

An Interface Communication Architecture.

Causal Model.

Causal Map.

Examples of Causal Rules.

Examples of Constraints.

Analysis of Examples of Posets.

Constraint Checking Becomes Part of the Collaboration.

13. Implementing Event Abstraction Hierarchies.

The Accessible Information Gap.

Event Abstraction Hierarchies, Revisited.

Induced Causality.

Abstraction Effect on Constraints.

Modifiability.

Bridging the Information Gaps.

Steps to Apply a Hierarchy to a Target System.

A Hierarchy for a Fabrication Process.

Personal Views.

Implementation.

Diagnostics.

14. Case Study: Viewing a Financial Trading System.

A Small Stock-Trading System.

The Information Gap for STS.

An Event Abstraction Hierarchy for STS.

Building the Event Abstraction Hierarchy.

Level 1.

Level 2.

Level 3.

Implementing Hierarchical Viewing for STS.

Three Steps toward Human Control.

Drill-Down Diagnostics.

Detecting Constraint Violations.

The Abstraction Effect.

Summary.

15. Infrastructure for Complex Event Processing.

Examples of Forms of Observed Events.

Interfacing CEP Infrastructure to Target Systems.

CEP Adapters.

CEP Runtime Infrastructure.

Infrastructure Interfaces and Components.

Functionality of the Interface.

Off-the-Shelf Infrastructure.

Event Pattern Languages.

Complex Event Pattern Matchers.

Quest for Scalability.

The Naive View of Pattern Matchers.

What Pattern Matchers Really Do.

Design Structure.

Rules Management.

Analysis Tools.

Summary.

Bibliography.
Index. 0201727897T04252002

Preface

Complex event processing (CEP) is a set of techniques and tools to help us understand and control event-driven information systems. And today, any kind of information system, from the Internet to a cell phone, is driven by events. What is a complex event? It is an event that could only happen if lots of other events happened.

For example, suppose you see a car you like at your favorite car dealership. That car is on the showroom floor only because a number of other events took place, events in the inventory control systems of the dealership and the manufacturer, shipping events, customs events at the port of entry, and so on. Of course, when you see exactly what you want in the showroom, you don't ask how or why! But if you don't see the model, make or color you want, and ask why not, then you'll get an explanation about allocation quotas, backlogs at the factory, or some other factors that affect events in the causal history leading up to the event you wanted.

This illustrates one of the ideas behind CEP. Events are related in various ways, by cause, by timing, and by membership. CEP applies to electronic information systems. It makes use of relationships between events to answer questions like, "is our system providing the correct level of service to our customers", "will our shipment arrive on time", and "is someone trying to steal our information". CEP adds a new dimension of event processing to what our event-driven information systems already do.

Why is there a need for CEP? Well, let's look at the situation, briefly.

Today's Information Society is founded upon gathering and sharing information. All of our organizations, commercial, government, and military, are dependent upon electronic information processing. Their foundational backbone is the kind of distributed computing system based on computer networks that is nowadays called the "information technology layer" (or IT layer) of the organization. The use of these systems has expanded rapidly over the past ten years to meet the increasing demands of automation, electronic commerce and the Internet explosion. Investment in technology has focussed on making IT systems faster, capable of handling larger and larger amounts of information, and able to collaborate with one another. We now live in the world of the open enterprise where commerce and information move across the boundaries of organizations, and nations. Our society has become dependent upon IT systems.

Less investment has been devoted to develop technology to solve the increasing problem of understanding what is happening in our IT systems. Whenever there is a crisis—a denial of service attack or a system failure—at first we don't understand what is going on, or how to fix it, and then in the aftermath we scramble for weeks to find out what caused it. We need to understand and control our critical information infrastructures better than that!

A lot of the information in IT systems is never recognized. Messages—or events—pass silently back and forth across our information systems as unrelated pieces of communication. They are a source of great power for when they are aggregated together, and correlated, and their relationships understood, they yield a wealth of information. A new technology is needed to harness the power of events in global information systems. This book is about such a technology.

A few words about CEP—what it is, and where it applies.

CEP consists of very simple techniques, a mix of old and new. Some of them are well known in other kinds of computer applications, such as rule-based systems in intelligent programs. Some of them are new techniques, such as tracking causal histories of events in large distributed computer systems. Or using patterns of events and event relationships, to recognize the presence of complex events that are signified by hundreds or thousands of simpler events in our IT systems. In CEP, new techniques are combined with well-known techniques in a unified framework.

An example of the kind of electronic complex event we are talking about is the completion of a financial transaction involving a bundle of financial contracts. Several merchant banks and brokerage houses may participate in the transaction. They use a global trading network. The event itself, the completion of the transaction, might be the result of hundreds of electronic messages and entries into several different databases around the world over a span of two or three days. These event don't necessarily happen in a nice linear order, one after the other. Some of them might happen simultaneously and independently of others, mixed in with events from other transactions. We can apply CEP to the trading network to recognize not only when that complex event happens, but more importantly, whether it is going to happen, or is getting off track and may not happen, and why!

CEP applies to a very broad spectrum of challenges in information systems. A short list includes (1) business process automation utilizing the Internet and electronic Marketplaces, (2) computer systems to automate the scheduling and control of anything from fabrication lines to air traffic, (3) network monitoring and performance prediction, and (4) detecting attempts to intrude into computer systems or attack them.

There is a fundamental reason for this broad applicability. It is simply because information systems are all driven by events. To be sure, each system, or application running on top of a system, depends upon different kinds of events. Network events are different from database events, which are different from financial trading events. But one of the major themes of CEP is that different kinds of events are related. CEP provides techniques for defining and utilizing relationships between events. CEP applies to any type of event that happens in a computer application or a network or an information system. In fact, one of its techniques lets you define your own events as patterns of the events in your computer system. CEP lets you see when your events happen. This is one way to understand what is going on in your system!

And that brings us to another point—flexibility. CEP allows users to specify the events that are of interest to them at any moment. Events of interest can be low level network monitoring alerts, or high-level enterprise management intelligence, depending upon the role and viewpoint of individual users. Different kinds of events can be specified and monitored simultaneously. And the specification of the events of interest, how they should be viewed and acted upon, can be changed on the fly, while the system is running.

The users of CEP can be human, or they can be automated processes. The processes that manage our enterprises are becoming more complex. Linear workflow processes that epitomize document processing in commercial transactions are not capable of managing the open electronic enterprise. In the future, enterprise management processes will be designed to incorporate complex event processing in order to get the kind of events they need to operate.

Now, a few words about the book itself and what the reader should expect. First of all, there are two parts to this book.

Part I is for a broad audience of people with an interest in various aspects of the Information Society, such as Electronic Commerce, the Internet, B2B collaboration, or generally, Electronic Information Processing. Part I deals with two questions about CEP. What it is for—that is, the kinds of problems in the Information Society that CEP can be applied to. And, what it is—a simplified view of CEP, the basic concepts and easy examples of applications.

Part I includes Chapters 1-7. The first four chapters describe the problems and issues in IT systems that CEP applies to. The next three chapters, 5-7, describe basic concepts of CEP such as what an "event" is, causal and timing relationships between events, patterns of events and event hierarchies, and how to apply them to solve the problems described earlier.

Part II consists of Chapter 8 onwards. It is intended for information systems specialists with some background in software. Part II presents how-to-build-it details and case studies of CEP applications. The goal of Part II is to describe what is needed to build applications of CEP that are capable of solving real world problems. It includes first of all, a detailed description of a complex event pattern language, reactive event pattern rules and event pattern constraints.

Secondly, Part II shows how to build solutions by using the event pattern rules and constraints to build event processing agents, and architectures of communicating agents. Part II includes case studies, as large and as detailed as one can fit in a chapter of a book.

The final chapter of this book deals with the question of how to develop an infrastructure for CEP. One can look around the event-driven applications being developed in the commercial world today, utilizing the power of distributed computing, the Internet and private networks. There is an almighty commercial struggle brewing for market share in the world of eMarketplaces and Electronic Commerce. It is quite predictable, considering the trends in middleware, the Java world, the .Net world, the security world, and so on, that CEP will be developed as a competitive advantage. This chapter deals with leveraging these developments to build infrastructure for CEP—now and quickly!

A word about references. This area of Internet technology is changing so quickly that any attempt to give comprehensive references would be outdated in six months. Not only that, but any less than complete set of references would be unfair to some. I assume that any reader has access to the Internet. So, they can search for current references to, e.g., "middleware" or "application server". So I have tended to include only a few references, either general references to Web sites or citations to seminal research papers that are not easily found.

At this current time in our society, any technology that attempts to view and control IT systems may be seen by some as conflicting with issues concerning privacy. In fact, CEP may provide a foundation for resolving some possible conflicts. However, I cannot deal with this topic here, and I do not.

Just a little history. CEP has grown out of a research project at Stanford on event -based simulation called the Rapide project. This research took place between 1990 and 2000. Out of Rapide came some early experiments in CEP applied to viewing small communicating systems built on commercial middleware, or applied to recognizing security threats in progress on the IT layer of a large university, where hackers love to play! These projects are documented on two Web sites:

http://pavg.stanford.edu/rapide/
http://pavg.stanford.edu/cep/
—David Luckham, Professor Emeritus,
Electrical Engineering, Stanford University


0201727897P10262001

Index

Click below to download the Index file related to this title:
Index

Updates

Submit Errata

More Information

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.

Overview


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information


To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information


Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children


This site is not directed to children under the age of 13.

Marketing


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information


If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information


Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents


California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure


Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact


Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice


We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020