CEP Round Table Discussion

Waters magazine have posted a pretty interesting "round table" discussion on the state of Complex Event Processing (CEP) featuring some of the vendors in this space and an implementor at HSBC. WARNING: Very long, but pretty comprehensive.

Some of the highlights:

Current Uses

• Front-office and some middle-office applications
• Goal: Externalize business rules from embedded code to a CEP intermediate language. (Nothing new here, Wall St firms have been trying to do this for years with expert systems and rules engines.)
  
• Managing the "lifecycle" of a trade. This may involve doing pre-trade processing - such as risk limits or satisfying legal requirements such as MiFID and RegNMS - or post-trade processing. Few people thought CEP implementations would be performant enough to do complex analytics that some algo trading systems do, but some said it was currently being used to do algo trading.
  

Inappropriate Uses

Things CEP is not suited for:

• Capturing "tick" data
• Correlating large amounts of data. This is more a problem with algorithms rather than a particular CEP implementation
  

HSBC Implementation

• HSBC have implemented their own CEP infrastructure using open source components (Esper?). Open source was a key feature because it allowed them to put together a new and possibly risky technology stack cheaply and quickly (6 months from design to production).
• One application using this CEP stack is Barracuda

Standards

• There are no current standards around CEP
• Most CEP implementations provide an SQL-like interface - see Marc's post for a good example of this. The panel agreed this was probably because SQL was familiar to many IT people, but not suitable for everyone (e.g. traders)
• One CEP vendor said their "rules GUI" provided a competitive advantage
• The HSBC representative preferred a UML-like GUI rather than SQL
• A barrier to implementing CEP is to convince business that it is ok to reimplement logic from their trusted embedded code to a new CEP language

Benchmarks

• No standard benchmarks yet, but the old chestnuts throughput and latency were mentioned
  

Future Uses

Although the "C" in CEP stands for "Complex", most current uses of CEP involve capturing relatively simple events. Some potential future uses for CEP:

• Increasingly complex real-time correlations. One example given was that of a hurricane reported heading for the Gulf of Mexico. Commodity traders typically react to such news by selling everything related to oil assets in the gulf. After they sell the traders then they analyze the news in more detail, determining exactly what assets could be effected. If they find assets that will be spared by the hurricane they start buying again. A CEP system might help the traders determine which assets will be effected more quickly than their rivals.
• Regulation are likely to require companies to store more and more data