One of the interesting subsets of Complex Event Processing is Event Stream Processing – dealing with event streams such as financial data feeds, or indeed any continuous stream of events of a similar type. In TIBCO BusinessEvents, any incoming event is treated as an individual event from the “event cloud” (for example via Java JMS / TIBCO EMS or TIBCO Rendezvous). Events are typically processed in rules by comparing them with other and past events (of any type), creating “complex” or abstract events, and are used to signify state changes in relevant entities.
The main event processing problem that is not ideal for production rule mechanisms is the very same problem that event stream processing engines are good at: aggregating similar event types per some property-based or time-based constraint. Such aggregates represent a type of complex event (or intermediate event) that can then be used as information, in comparisons with other events and data, in rules. Not surprisingly, most of the custom Event Stream Processing engines marketed today use variations on the theme of the standard conventional relational database query language – SQL – for defining these aggregations [*1].
So what are the aggregation operations we might want to carry out on events (and event objects) of a similar type?
1. Snapshot
The most obvious query type is a snapshot against the historic event store. This is exactly the same as conventional database queries:
select s.symbol, avg(s.price) avgprice
from /ConceptModel/StockTick s
group by s.symbol
order by avgprice;
This is “executed” on demand, for example when a particular event occurs and we need the current average price list for use in some rules.
2. Simple Continuous View
The obvious flaw with simple snapshots (and, ergo, systems architected around database platforms) is that their results are only known to be correct at the point in time they are executed. In event-driven systems, new events may arrive at any time, and our CEP and decision rules need to refire when the aggregate results change. So snapshot queries can also be executed as a snapshot followed by a continuous query: whenever changes in the selected events cause a changed result, an appropriate update event can be sent to the appropriate rules together with the new aggregate result.
For example, consider this slight variation on the earlier example:
select s.symbol, avg(s.price) avgprice
from /ConceptModel/StockTick s
group by s.symbol
order by avgprice desc {limit : first 2};
This returns the 2 stock symbols with the highest average price… and when executed as a continuous query, repeats this whenever one of the top 2 stocks change.
3. Continuous View based on a specified window
A refinement on the simple continuous query is defining it to return a resultset based on some dynamic constraint such as period of time, or number of events. Such queries can be “sliding windows” (with the resultsets mimicking a First In First Out queue), “tumbling windows” (with full resultsets being emptied completely before the next window), or a “time window” (based on some time constraint such as the last n milliseconds).
Resultset notifications can be on new entries to the resultset, or when events leave the resultset. For example:
select coldpizza from /PizzaOrderEvent {policy: maintain last 45 minutes; emit: dead} coldpizza;
—
In TIBCO BusinessEvents, queries are defined and executed dynamically as required and can be fully parameterized to allow for re-use (and, if desired, an element of machine learning such as adjusting the time window of a query based on some other parameters). In addition, the full set of functions (including custom extensions) available in production rule execution can be included in query statements.
Overall, event stream / dynamic data queries extend a standard and common IT language (SQL) in order to define aggregates of events and data to help support dynamic, event-driven business systems.
Notes:
Past discussions of this ilk have included the value of state and production rules in Complex Event Processing.
[1] See also past research on “data stream technology” (Stanford Stream, Brandeis/Brown/MIT Aurora, Purdue’s Nile, etc) that led to Continuous Query Languages and (indirectly) to various event stream processing vendors in the past decade.
