CEP Round Table Discussion

Waters magazine have posted a pretty interesting "round table" discussion on the state of Complex Event Processing (CEP) featuring some of the vendors in this space and an implementor at HSBC. WARNING: Very long, but pretty comprehensive.

Some of the highlights:

Current Uses

• Front-office and some middle-office applications
• Goal: Externalize business rules from embedded code to a CEP intermediate language. (Nothing new here, Wall St firms have been trying to do this for years with expert systems and rules engines.)
  
• Managing the "lifecycle" of a trade. This may involve doing pre-trade processing - such as risk limits or satisfying legal requirements such as MiFID and RegNMS - or post-trade processing. Few people thought CEP implementations would be performant enough to do complex analytics that some algo trading systems do, but some said it was currently being used to do algo trading.
  

Inappropriate Uses

Things CEP is not suited for:

• Capturing "tick" data
• Correlating large amounts of data. This is more a problem with algorithms rather than a particular CEP implementation
  

HSBC Implementation

• HSBC have implemented their own CEP infrastructure using open source components (Esper?). Open source was a key feature because it allowed them to put together a new and possibly risky technology stack cheaply and quickly (6 months from design to production).
• One application using this CEP stack is Barracuda

Standards

• There are no current standards around CEP
• Most CEP implementations provide an SQL-like interface - see Marc's post for a good example of this. The panel agreed this was probably because SQL was familiar to many IT people, but not suitable for everyone (e.g. traders)
• One CEP vendor said their "rules GUI" provided a competitive advantage
• The HSBC representative preferred a UML-like GUI rather than SQL
• A barrier to implementing CEP is to convince business that it is ok to reimplement logic from their trusted embedded code to a new CEP language

Benchmarks

• No standard benchmarks yet, but the old chestnuts throughput and latency were mentioned
  

Future Uses

Although the "C" in CEP stands for "Complex", most current uses of CEP involve capturing relatively simple events. Some potential future uses for CEP:

• Increasingly complex real-time correlations. One example given was that of a hurricane reported heading for the Gulf of Mexico. Commodity traders typically react to such news by selling everything related to oil assets in the gulf. After they sell the traders then they analyze the news in more detail, determining exactly what assets could be effected. If they find assets that will be spared by the hurricane they start buying again. A CEP system might help the traders determine which assets will be effected more quickly than their rivals.
• Regulation are likely to require companies to store more and more data
  

GMail upgrade breaks Greasemonkey Ad Remover scripts, adds API for Greasemonkey

On October 29 GMail did an upgrade that changed many of the internal HTML elements used to display email. i.e. many of then element id's and class names changed. Unfortunately this broke some of my favorite Greasemonkey scripts such as GMail Ad Remover (removes the Sponsored Links) and GMail Full Width (removes Sponsored Links and widens the conversation area by moving buttons around).

Some of the other fans of the ad remover scripts also noticed it had been broken. Very interestingly the GMail developers had this to say:

While we (like most web services) don't officially support third-party extensions like Greasemonkey scripts, we realize that some of our most active users want to write and run them. Because these scripts directly modify a web service's code rather than using a stable API, they tend to be fragile to even small changes in a web app's code (and can even create bugs in the web app itself).

To make this easier on our Greasemonkey users, we've recently added an experimental Gmail/Greasemonkey API that should make these types of scripts easier to write and more robust to code changes.

Wow! Of course Google are not under any obligation to support third-party extensions (after all you can't call HTML page layouts an "external API"). It is pretty impressive that a company as big as Google are attempting to provide a compatibility layer for a third-party extension they have no financial interest in.

So what does a Greasemonkey script that hides GMail ads and uses this new API look like? This does the trick:

gmailadremover.user.js

// ==UserScript==
// @name           Gmail Ad Remover
// @namespace      http://robertmaldon.blogspot.com/gmailadremover
// @description    Remove the Sponsored Links from GMail
// @include        http://mail.google.com/*
// @include        https://mail.google.com/*
// ==/UserScript==

window.addEventListener('load', function() {
if (unsafeWindow.gmonkey) {
 unsafeWindow.gmonkey.load('1.0', function(gmail) {
   function removeAds() {
     if (gmail.getActiveViewType() == 'cv') {
       var sponsored = gmail.getConvRhsElement().lastChild;
    sponsored.style.display = 'none';
     }
   }
   gmail.registerViewChangeCallback(removeAds);
   removeAds();
 });
}
}, true);

While the ads are hidden, however, the table column that held the ads - and the "Print all", "Expand all", etc buttons - still take up a lot of space. So a quick and dirty GMail Full Width script that hides the whole column (including the buttons unfortunately) looks like:

gmailadcolumnremover.user.js

// ==UserScript==
// @name           Gmail Ad Column Remover
// @namespace      http://robertmaldon.blogspot.com/gmailadcolumnremover
// @description    Remove the Sponsored Links table column from GMail
// @include        http://mail.google.com/*
// @include        https://mail.google.com/*
// ==/UserScript==

window.addEventListener('load', function() {
if (unsafeWindow.gmonkey) {
 unsafeWindow.gmonkey.load('1.0', function(gmail) {
   function removeAdColumn() {
     if (gmail.getActiveViewType() == 'cv') {
    var sponsored = gmail.getConvRhsElement().lastChild;
    var sponsoredColumn = sponsored.parentNode.parentNode;
    sponsoredColumn.style.display = 'none';
    var convTable = gmail.getConvRhsElement().parentNode.parentNode.parentNode;
    convTable.width = '100%';
     }
   }
   gmail.registerViewChangeCallback(removeAdColumn);
   removeAdColumn();
 });
}
}, true);

When I get some more time I'll have to explore the power of this new API - and write a proper full width script that moves the buttons to a different location!

UML is dead, long live LSD

As part of a project hand-over I've recently done at a client there was the inevitable set of "standard" documents to write, such as a Design Document and a Troubleshooting guide.

There was no standard template for the Design Document, so amongst other things I included some UML class diagrams.


I was not specifically asked to include UML diagrams, but there was no objection to including them.

On reflection I haven't been asked by a client to specifically do UML diagrams for about 2 years now. Is this an indicator that UML is dead?

If UML is on its last legs what has replaced it? Seems to be what we had before Booch, Rumbaugh and the whole UML craze came along: ah-hoc component diagrams and sequence diagrams. (Although component and sequence diagrams are part of UML the UML versions have a particular look and convention to them.) Specs also seem to be getting thinner. I think the mainstreaming of Agile and Lean Software Development (LSD) ideas are taking effect.

btw, if you are using Eclipse there are a bunch of UML plugins that can help you draw UML diagrams. Most are a real pain to install. AmaterasUML, however, is very easy to install and has just enough features to make it useful (e.g. drag a Java class from the Java view and just drop it on to an AmaterasUML diagram view).

Minimial rsync/ssh for Windows

Every now and again I'm required to synchronize files between Windows and UNIX, usually a Windows desktop and UNIX server. Some of the commons tools for this task include a Samba Samba / robocopy combination, or installing the UNIX-like cygwin tools to use rsync/scp.

I personally think for this purpose cygwin is overkill and robocopy is slow and not very intuitive to use. With this bias in mind I was happy to stumble upon this article on installing a minimal rsync/ssh package on Windows. I must try this package out next opportunity!

RedHat buys then open sources AMQP implementation

How did I miss this event?

Previously I've blogged about Advanced Message Queue Protocol (AMQP), an effort by JPMorgan to commoditize messaging and break the back (i.e. save on the hugely expensive licensing costs) of the messaging behemoths Tibco and IBM.

According to this article RedHat bought an AMQP implementation from JPMorgan, improved it, then gave the implementation to the Apache Foundation as the QPid project. Currently there is a Java and C++ broker with clients in C++, Java (JMS), Ruby, Pyhton and C# for .NET.

Red Hat has the ambitious goal of enabling a server to process 1 million messages per second.

I guess JP didn't have an interest in maintaining the AMQP implementation, but RedHat are not exactly cheap when it comes to maintenance fees (but so far cheaper than Tibco and IBM).

Update: Microsoft have joined the AMQP working group!

When Network Cables Attack

I bet you this is Time-Warner's data center...

(Picture from http://kevinremde.members.winisp.net/images/yellow_wall.jpg)

Swing gets even slower by using Vector Graphics

In previous posts I've lamented how Swing had great potential to capture the hearts and minds of GUI developers, but stubbornness on not using native widgets and a bad implementation means they have forfeited GUI applications to competitors like .NET. Most new thick GUIs that our financial clients ask us to develop are .NET GUIs (C# is the language of choice).

I was reminded recently of the pitfalls of the Swing draw-each-widget-pixel-by-pixel approach when I used a simple GUI like JConsole to access an application that I was JMX-ising:


Horrible refresh problem. Swing applications seem to be much more prone to this than native applications.

Can I blame this on the pixel-by-pixel approach, or just Swing's implementation of it? One other implementation of pixel-by-pixel that I played with a few years ago is Thinlet, a remarkably tiny GUI toolkit. I have never put a Thinlet application into production (only recently have clients gotten comfortable with the LGPL license), but I was impressed with the speed of it.

Recently I read an article Nimbus, a new sexy look-and-feel that the Sun Swing engineers are working hard on. When I read that Nimbus used vector graphics I cringed. Even with faster CPUs I bet the performance will still be similar to the current pixel-by-pixel approach. i.e. sucks

Don't get me wrong. Having a vector graphics option is good for some GUIs, but I really wish Sun (or whoever ends up driving the GUI side of Java) would also standardize a fast GUI toolkit. SWT is one option. Thinlet is another.

It is also interesting to note that Android, Google's Java-based framework for developing mobile phone applications, does not include any of the standard AWT or Swing libraries, instead using OpenGL C/C++ bindings.

Shame Sun, Shame for missing the opportunity to take over the desktop.

Dynamically add to the Eclipse JUnit classpath

When you run JUnit tests inside Eclipse something that may bite you is the fact that the compile classpath and runtime classpath are separate.

Let's say your Eclipse project directories are layed out something like the following:

PROJ_ROOT

• bin
• conf
• src
• test

where src contains your core application source code and test contains your JUnit tests.

When a JUnit test runs, by default the base run directory and the classpath root is the PROJ_ROOT directory. Therefore, if your unit test loads a file relative to the the PROJ_ROOT, say:

new FileInputStream("conf/context.xml");

or

Thread.currentThread().getContextClassLoader().getResourceAsStream("conf/context.xml");

it will work. If, however, you've told Eclipse that conf is a "src" directory (i.e. part of the compile classpath) and then try to load a file from somewhere in the classpath from your unit test like so:

Thread.currentThread().getContextClassLoader().getResourceAsStream("context.xml");

it will fail.

So what can you do to change the classpath for the unit tests that need it?

You can manually add the conf directory to the runtime classpath for just those unit tests that need them, but unfortunately so does everyone else in your team.

A cute trick is to change the thread classloader to a URLClassLoader, whose constructor allows you to add URLs (including directories) to the runtime classpath:

ClassLoader currentThreadClassLoader
 = Thread.currentThread().getContextClassLoader();

// Add the conf dir to the classpath
// Chain the current thread classloader
URLClassLoader urlClassLoader
 = new URLClassLoader(new URL[]{new File("conf").toURL()},
                      currentClassLoader);

// Replace the thread classloader - assumes
// you have permissions to do so
Thread.currentThread().setContextClassLoader(urlClassLoader);

// This should work now!
Thread.currentThread().getContextClassLoader().getResourceAsStream("context.xml");

To give credit where it is due the above is a slightly simplified version of a solution from my colleague Randy who came up with it when we worked together on a project earlier this year. His solution makes the assumption that the JVMs system classloader is a URLClassLoader (may not be true for all JVMs) and uses a bit of reflection magic to add to the classpath:

public void addURL(URL url) throws Exception {
  URLClassLoader classLoader
         = (URLClassLoader) ClassLoader.getSystemClassLoader();
  Class clazz= URLClassLoader.class;

  // Use reflection
  Method method= clazz.getDeclaredMethod("addURL", new Class[] { URL.class });
  method.setAccessible(true);
  method.invoke(classLoader, new Object[] { url });
}

addURL(new File("conf").toURL());

// This should work now!
Thread.currentThread().getContextClassLoader().getResourceAsStream("context.xml");

When Teachers Attack - Caught On Mobile Phone Camera

My own experience of High School showed me teachers were a mixed bag - some excellent, some very average and some very lazy. Occasionally good teachers had bad moments.

I've got mixed feelings about YouTube's 7 Scariest Teachers, a selection of teacher rants caught by students with mobile (cell) phone cameras. Should teachers, who are already struggling to teach unmotivated students, be exposed to public ridicule like this? On the other hand this could help weed out people who shouldn't be teaching.

What keywords do recruiters search for?

It's well known that recruiters / job agents do a lot of filtering of resumes using keyword searches. (I get a lot of spam from recruiters based on a technology I might have used once or twice in my career that I mentioned on my resume at some point in the past.)

But what keywords are hot right now? Marketing job website MktgLadder have made available a regularly updated Top 100 Keywords searched for by recruiters. I would like to see this feature added to some of the more IT-centric job sites like Dice.