Steve Nimmons of Atos Origin looks at complex event processing.
There is an inherent complexity in understanding the relationship between what can appear to be seemingly unconnected events occurring in real or near-real time. I make the temporal distinction as there are sophisticated business intelligence and data mining solutions for pattern or trend discovery in previously captured business information.
These are proven and do a very solid job in specific circumstances. There are also interesting extensions emerging in terms of mash-ups that augment and enrich more fully-featured end-user driven business intelligence solutions. Analysing events in real-time can be exceptionally informative, adding to the overall utility of business intelligence and providing a mechanism for business processes to react advantageously.
I describe complex event processing (CEP) as gold panning the elemental binary soup. CEP is trying to amplify the signal of interesting information against the noise of your event driven architecture. What is interesting is generally termed a 'concept' and CEP is all about trying to instantiate concepts from the flow of events.
CEP is typically achieved using forward chaining rules engines, themselves based on the RETE algorithm. There are a number of examples of good COTS products, I myself had the privilege of working on one prototype system that has now been successfully productised.
The business problem we faced was the correlation of events in a distributed mobile telecoms network. Tracing failures throughout the user experience was complex and put a large burden on operations and customer service representatives (CSRs).
The solution was to decode large volumes of real-time data from RADIUS servers, WAP servers, MMS, MML, SMS, micropayment systems et al, and hunt for failure patterns in the datasets (and present these in a simple consumable format). The result was exceptional operational saving, increased user satisfaction and greater insight into systems failures (which were then addressed).
Having 'done battle' with early systems I table the following implementation recommendations:
Solutions and industries in which I see increased CEP interest and uptake include: transport, financial services (fraud detection) and fault monitoring systems (cross industries). The latter case is very interesting and systems are being developed that seek to provide early warning of developing faults that generate event streams with tell-tale characteristics.
The definition of 'tell-tale' is of course rather slippery, with practically limitless variation between circumstances. This is at the heart of what makes CEP both interesting and challenging.