A colleague asked for some comments on the terms “CEP” and “ESP” – and in particular why, if a customer already had a stream processing product, they might still need some alternative or non-stream technology for certain other event processing duties…
1. There is growing industry consensus on the “CEP” and “ESP” terminology. “Event stream processing” is for things like market data streams (usually a high ratio of event throughput vs numbers of queries), whereas “complex event processing” is a superset term that also covers event-by-event processing and aggregation (e.g. on potentially out-of-order events from a variety of sources – often with large numbers of rules or business logic).
Sources:
a. The EPTS Glossary by most of the CEP vendor community covers these terms. In particular it defines Event Stream Processing as the processing of in-order events.
b. The “inventor” of the term CEP, David Luckham, provides an excellent differentiation on his web site.
c. Analyst comments on this differentiation by Bloor .
d. 3rd party blog comments by Opher Etzion , by Jack vanHoof , and by Tim Bass .
2. Generally “stream processing” processes event streams as “sets” and does “set type” operations, typically using “continuous queries” (SQL-type queries that operate over time and buffer windows). The wider “complex event processing” term additionally covers other mechanisms like ECA rules, production rules, and so forth. By definition, ESP products are therefore also CEP products in the same way that both Edsels and Bugattis are all cars.
While streaming query engines are very useful technologies they are not optimal for all CEP problems – in particular they have different characteristics regarding, for example, comparing out-of-order or out-of-stream events, applying decisions and reactions to event patterns, and so on. For this reason multiple CEP language classes have evolved, typically described as queries, rules and procedural approaches (to event pattern detection). This is also why TIBCO BusinessEvents explicitly supports state models and production rules as well as continuous queries, and is architected to allow additional Event Processing Languages to be added in future if desired.
Sources:
a. TIBCO Blog on Queries based on TIBCO’s coverage of ESP. Other ESP vendors are commented on in another post.
b. EPTS Language Working Group is defining some differentiations here but have not published their work yet: we blogged on a preview at a recent conference.
3. Because ESP is but a specialised subset of CEP, some people may have tried to make “ESP” synonymous with “CEP”. This remains a very grey area as while event stream processing using queries is a very useful capability (i.e. event set handling and detection), so is event-by-event handling and correlation using, say, inference rules.
So… in most enterprises there are usually multiple use cases for multiple types of CEP that are best handled by multiple paradigms (such as specialist ESP, event-driven business processes, rule-driven event processing, event-based business rules, event-driven analytics, etc). One should no more expect a large company to rely on a single CEP paradigm as it would on a single computer hardware technology.
