Emacs.NET coming soon?

Why is Microsoft writing a .NET version of the good 'ol UNIX text editor Emacs?

I suspect it will be not a direct clone of Emacs, but like Emacs will be a text editor that is largely written in and customized by .NET scripting languages (IronRuby, VB, etc), maybe even integrated with PowerShell.

Will it be open source?

Update: The mystery has become a little less mysterious. The editor mentioned above will be called "Intellipad". It is apparently going to be part of Microsoft's SOA offering, code named "Oslo". Another major component of Oslo is a declarative “textual modeling language” called "D", which will be based on XAML. Dunno how a modeling language like D could be based on a markup language like XAML...

Last Minute Geek Xmas Gifts

Looking for a last minute xmas gift for that special geek someone? Are you a geek with lots of gift cards to get rid of? Here are my suggestions for 2007:

8. Reindeer Computer Carton

Feeling creative? Don't know what to do with all of that packaging that came with your new computer? Make a reindeer out of it!

7. Disembodied Remote Controlled Hand
Just like Thing from the Adams Family. Get your own remote control hand that crawls across the floor on its fingers!

6. Toilet Aquarium

Turn your boring toilet into a Fish 'N Flush "nautical wonderland"! Fish not included.

5. Fun Toilet Paper
You're not going anywhere until you wipe. Revenge Toilet Paper looks like the real thing but cannot be torn!
Give politicians back the same stuff they give you with Political Toilet Paper.

4. A Chair That Follows You

For the ultimate in lazy the “Take-A-Seat” is a chair concept that follows you wherever you go!

3. Wi-Fi Detector Shirt

Display the current wi-fi signal strength to yourself and everyone around you with this stylish Wi-Fi Detector Shirt. The glowing bars on the front of the shirt dynamically change as the surrounding wi-fi signal strength fluctuates!

2. iPod Accessories

Let your favorite tunes clean your teeth with Rock My Teeth.

Turn your dog into a boom box with the dog-jacket iPod dock.

Checkout some other iCrap.

1. Motorized Cooler

Ever drive to a park or friends house for a BBQ and then discover you have to carry a heavy cooler full of food and drink to some far off area? No more! Let your Cruzin Cooler take you and your food to the party in style!

Changing Windows Process Scheduling with ProcessTamer

For years I've been disappointed with the Windows process scheduler. It is so easy for a process to suddenly take up all of the CPU and completely "freeze" your PC for minutes at a time.

Fair scheduling is something high-end UNIX variants like Solaris have dealt with successfully for a while now, while Linux is still coming to terms with.

An interesting freeware tool I've been trying out recently is ProcessTamer, a Windows utility that runs in your system tray and constantly monitors the cpu usage of other processes. When it sees a process that is overloading your cpu, it reduces the priority of that process temporarily, until its cpu usage returns to a reasonable level. Now my PC doesn't become completely unresponsive for long periods of time, but on the downside some applications like Firefox take longer to start up.

Sad that I have to consider using tools like ProcessTamer, but so far it has done a pretty decent job.

Virtualization == Increased Latency?

An interesting quote that caught my eye while reading about the NYSEs replacement of UNIX servers with Linux (nothing really new there since many of my financial clients started on that 5+ years ago) was this:

One technology that the NYSE isn't adopting so eagerly is server virtualization, which comes with a system latency price that Rubinow said he can't afford to pay. In a system that is processing hundreds of thousands of transactions per second, virtualization produces "a noticeable overhead" that can slow down throughput, according to Rubinow. "Virtualization is not a free technology from a latency perspective, so we don't use it in the core of what we do," he said.

Charles King, an analyst at Pund-IT Inc. in Hayward, Calif., believes there is a broader concern among IT managers about virtualization overhead and its impact on transaction processing. "It's one of the reasons why even the staunchest advocates of x86 virtualization recommend extensive testing prior to moving systems into production," King said.

I haven't heard this particular concern until now, but I can imagine it to be a big one since one thing you will hear talked about again and again and again if you work on trading systems is "low latency".

So what can be done to reduce latency in a virtualized environment? Depends on how much you're prepared to spend.

• Software virtualization like VMWare or Parallels is likely to incur some extra latency above native-run applications, even though they take advantage of recent hardware virtualization hooks. In theory the more powerful the hardware the lower the latency.


• The top of the line virtualization (i.e. very expensive) offering is probably still Azul. This is a hardware platform for running virtual machines across a pool of Azul "appliances". The hardware has been optimized for application context switching, large memory heaps, etc. Azul like to call their offering "application virtualization" and products like VMWare are called "OS-level virtualization".

Using Web 2.0 to find the nearest toilet

I was excited to see the recent launch of MizPee, a website with SMS support that can help you find the nearest toilet in your hour of greatest need.

MizPee serves up a list of nearby toilets, how far away the toilet is, a rating and whether or not there is a charge. Other details may include disabled access, whether the restroom includes a diaper-changing station. Users can also leave comments.

There were many critical comments about the site, but having been a tourist I can say that not every city has a Starbucks or McDonalds on every block and not every restaurant is happy to let people walk in off the street and use their loos. While one comment said "web 2.0 just jumped the shark" I've got to say that it is a good thing the barrier to entry for a webapp is now so low that a niche application like this can exist.

A niche idea I've had for a while: In the part of Brooklyn I live there is the joy of alternate side street parking, which means I have to move my car at least once a week. I'de love to put together a webapp that could help me locate the nearest parking spot. Big challenges in keeping that info up to date since spots come and go very quickly.

Crowdsourcing your next development project

Wikipedia defines crowdsourcing as:

The act of taking a task traditionally performed by an employee or contractor, and outsourcing it to an undefined, generally large group of people, in the form of an open call. For example, the public may be invited to develop a new technology, carry out a design task, refine an algorithm or help capture, systematize or analyze large amounts of data.

Although it has had some success I wonder can it work for developing large, complex applications like many of the financial applications I work on for a living. I can see it working for fairly generic applications (e.g. many webapps), but complexity takes a lot of managing. Many of the finance companies would also be reluctant to make requirements publicly available, exposing their intellectual property to their competitors.

However, a large chunk of any application is plumbing code. This article on crowdsourcing demonstrates how one company opened up about half of an application to crowdsourcing and is committed to do th rest of the integration work:


In Baltimore-based Constellation Energy's case, the $19.3 billion energy company didn't stage a completely open call; rather, it worked with TopCoder , a Connecticut-based company that stages regular coding competitions, ranks developers who compete and then makes this talent available to businesses that need systems built, also through a competition-based model. TopCoder currently has about 130,000 members from more than 200 countries.

A TopCoder project manager assessed the needs of Constellation Energy's commodities group, broke up the system design into dozens of small components and released about half of those component requirements to member developers, who could send in their best coding effort. (Constellation decided to build some of the components in-house.)

Submissions -- which continue to roll in -- are rated using a standardized scorecard, and winners are rewarded anywhere from $500 to close to $2,000. When all the components are complete, TopCoder will work with Constellation to integrate them into a functional system.


It will be interesting to see how this project turns out.

I had a client a couple of years ago that was not happy with the quality of code coming back from an outsourcing vendor and had the idea that he would set up a team in NY to integrate (and fix!) code coming back from this vendor, but I don't see why this integration team couldn't integrate the best code from a few competing vendors!

Google Mobile Maps pin points your location without GPS

Google continues to innovate. Last week they released an update to their Google Maps for mobile suite with an application called "My Location", a technology which uses cell tower ID information to provide users with their approximate location. Good for people with ancient cell phones sans GPS like me! (video explanation here)

Instaling and Configuring the Perforce Eclipse Plugin

A few clients I've had this year use Perforce for source control. I didn't find the official Perforce Eclipse plug-in installation notes detailed enough, so here is my version:

Prerequisites

1. If you haven't got the Perforce client already installed (the Eclipse plug-in uses the p4.exe command-line tool) then download it from here and install.

2. Create a Perforce workspace for your project.

Eclipse Install and Configuration

1. If you are behind a corporate firewall configure Eclipse to use the firewall HTTP proxy. Proxy configuration steps vary with the version of Eclipse, but for Eclipse 3.1 it is :

a. Windows > Preferences > Internet > Proxy Settings
b. Check Enable Proxy
c. Type in the Proxy host and Proxy port
d. Click OK

2. Add the Perforce plug-in remote site

a. Help > Software Updates > Find and Install
b. Click “Search for new features to install” > Next
c. Click “New Remote Site”
d. In the dialog box enter

Perforce Plugin

for “Name” and

http://www.perforce.com/downloads/http/p4-wsad/install/

for “URL” > OK
e. Click Finish

3. Download the plug-in


a. Check the “Perforce Plugin” checkbox > Finish

b. Check the “Perforce Plugin” checkbox > Next

c. Click “I accept the terms in the license agreement” radio button > Next

d. Accept the default plug-in install location by clicking Finish

e. Click Install All

f. Click Yes when asked if you want to restart Eclipse

4. Enable label decorations

a. Windows > Preferences > General > Appearance > Label Decorations
b. Check “Perforce” > OK

5. Configure your favorite Perforce Label Decorations

a. Windows > Preferences > Team > Perforce > Label Decorations
b. Select the decorators and decorator locations from the "File Decoration Icons" combo boxes
c. Click OK

6. Configure Eclipse to point to the p4 executable

If the Perforce p4.exe command-line tool is not in your PATH then you will need to explicitly set the location of it.

a. Windows > Preferences > Team > Perforce
b. Click the "Location" radio button
c. Enter the full path to the p4.exe in the text box e.g. C:\Program Files\Perforce\p4.exe
d. Click OK

7. Configure your project to point to your Perforce workspace

a. In the Eclipse project hierarchy right-click the name of your project > Team > Share Project…


b. Click “Perforce” > Next

c. Enter the PERFORCE_SERVER_NAME:PERFORCE_SERVER_PORT for “Port”, the Perforce user name for “User” and Perforce workspace for “Client Workspace” > Finish

Recovering Your Eclipse Workspace

One situation you're likely to come across when using the Perforce plug-in is this: if Eclipse does not shut down properly (e.g. Eclipse crashes or your PC did a forced reboot) then chances are pretty good your Eclipse workspace will be corrupted.

The symptom of this is that when you try to start Eclipse the splash screen will appear, then disappear and then nothing happens. i.e. Eclipse fails to start. You can verify the cause by looking for Perforce plug-in failure messages in the WORKSPACE_DIR/.metadata/.log file.

There are two solutions to this:

1. Re-create your workspace from scratch, which can take a while.
2. Manually uninstall and re-install the Perforce plug-in.

To manually uninstall and re-install the Perforce plug-in do the following:

i. Exit Eclipse if it is running.
ii. Move the sub-directories beginning with com.perforce found in ECLIPSE_INSTALL_DIR/plugins to a temporary directory (e.g. c:\Temp). In the version of the Perforce plug-in I installed these sub-directories are named

• com.perforce.p4api_2006.2.4136
• com.perforce.p4wsad_2006.2.4136
• com.perforce.team.core_2006.2.4136
• com.perforce.team.ui_2006.2.4136

iii. Start Eclipse, let it load the workspace, then exit Eclipse normally
iv. Move the Perforce plugin sub-directories back to ECLIPSE_INSTALL_DIR/plugins
v. Start Eclipse
vi. Re-associate the project with the Perforce workspace as per step 7 above

Konsultant Number One

My first Kid Consultant (Konsultant) Jacqueline joined the ranks of the Maldon Family Consultancy (MFC) last week.

Architect Jackie helps JSP Monkey daddy debug a webapp

If you would like to hire her then here is a summary of her resume (following the industry standard of experience exaggeration):

Roles: New Media Architect, iCommerce eGuru, Hungry Baby
Certifications: Scrum Master, Lean Wizard
Experience:

• 25 years Principles of Object Oriented Programming (POOP)
• 15 years B2B, C2C and D2D
• 13 years Enterprise Service Bus (ESB), Complex Event Processing (CEP)
  
• 12 years Ruby beyond Rails (RbR)

Patterns: Singleton (is there any other pattern than singleton?)
Technologies:

• Awesome Modeling Language (AML) 3.0, Spring 8.0, Hibernate 9.3, Oracle 53.6.0.2
  

CEP Round Table Discussion

Waters magazine have posted a pretty interesting "round table" discussion on the state of Complex Event Processing (CEP) featuring some of the vendors in this space and an implementor at HSBC. WARNING: Very long, but pretty comprehensive.

Some of the highlights:

Current Uses

• Front-office and some middle-office applications
• Goal: Externalize business rules from embedded code to a CEP intermediate language. (Nothing new here, Wall St firms have been trying to do this for years with expert systems and rules engines.)
  
• Managing the "lifecycle" of a trade. This may involve doing pre-trade processing - such as risk limits or satisfying legal requirements such as MiFID and RegNMS - or post-trade processing. Few people thought CEP implementations would be performant enough to do complex analytics that some algo trading systems do, but some said it was currently being used to do algo trading.
  

Inappropriate Uses

Things CEP is not suited for:

• Capturing "tick" data
• Correlating large amounts of data. This is more a problem with algorithms rather than a particular CEP implementation
  

HSBC Implementation

• HSBC have implemented their own CEP infrastructure using open source components (Esper?). Open source was a key feature because it allowed them to put together a new and possibly risky technology stack cheaply and quickly (6 months from design to production).
• One application using this CEP stack is Barracuda

Standards

• There are no current standards around CEP
• Most CEP implementations provide an SQL-like interface - see Marc's post for a good example of this. The panel agreed this was probably because SQL was familiar to many IT people, but not suitable for everyone (e.g. traders)
• One CEP vendor said their "rules GUI" provided a competitive advantage
• The HSBC representative preferred a UML-like GUI rather than SQL
• A barrier to implementing CEP is to convince business that it is ok to reimplement logic from their trusted embedded code to a new CEP language

Benchmarks

• No standard benchmarks yet, but the old chestnuts throughput and latency were mentioned
  

Future Uses

Although the "C" in CEP stands for "Complex", most current uses of CEP involve capturing relatively simple events. Some potential future uses for CEP:

• Increasingly complex real-time correlations. One example given was that of a hurricane reported heading for the Gulf of Mexico. Commodity traders typically react to such news by selling everything related to oil assets in the gulf. After they sell the traders then they analyze the news in more detail, determining exactly what assets could be effected. If they find assets that will be spared by the hurricane they start buying again. A CEP system might help the traders determine which assets will be effected more quickly than their rivals.
• Regulation are likely to require companies to store more and more data
  

GMail upgrade breaks Greasemonkey Ad Remover scripts, adds API for Greasemonkey

On October 29 GMail did an upgrade that changed many of the internal HTML elements used to display email. i.e. many of then element id's and class names changed. Unfortunately this broke some of my favorite Greasemonkey scripts such as GMail Ad Remover (removes the Sponsored Links) and GMail Full Width (removes Sponsored Links and widens the conversation area by moving buttons around).

Some of the other fans of the ad remover scripts also noticed it had been broken. Very interestingly the GMail developers had this to say:

While we (like most web services) don't officially support third-party extensions like Greasemonkey scripts, we realize that some of our most active users want to write and run them. Because these scripts directly modify a web service's code rather than using a stable API, they tend to be fragile to even small changes in a web app's code (and can even create bugs in the web app itself).

To make this easier on our Greasemonkey users, we've recently added an experimental Gmail/Greasemonkey API that should make these types of scripts easier to write and more robust to code changes.

Wow! Of course Google are not under any obligation to support third-party extensions (after all you can't call HTML page layouts an "external API"). It is pretty impressive that a company as big as Google are attempting to provide a compatibility layer for a third-party extension they have no financial interest in.

So what does a Greasemonkey script that hides GMail ads and uses this new API look like? This does the trick:

gmailadremover.user.js

// ==UserScript==
// @name           Gmail Ad Remover
// @namespace      http://robertmaldon.blogspot.com/gmailadremover
// @description    Remove the Sponsored Links from GMail
// @include        http://mail.google.com/*
// @include        https://mail.google.com/*
// ==/UserScript==

window.addEventListener('load', function() {
if (unsafeWindow.gmonkey) {
 unsafeWindow.gmonkey.load('1.0', function(gmail) {
   function removeAds() {
     if (gmail.getActiveViewType() == 'cv') {
       var sponsored = gmail.getConvRhsElement().lastChild;
    sponsored.style.display = 'none';
     }
   }
   gmail.registerViewChangeCallback(removeAds);
   removeAds();
 });
}
}, true);

While the ads are hidden, however, the table column that held the ads - and the "Print all", "Expand all", etc buttons - still take up a lot of space. So a quick and dirty GMail Full Width script that hides the whole column (including the buttons unfortunately) looks like:

gmailadcolumnremover.user.js

// ==UserScript==
// @name           Gmail Ad Column Remover
// @namespace      http://robertmaldon.blogspot.com/gmailadcolumnremover
// @description    Remove the Sponsored Links table column from GMail
// @include        http://mail.google.com/*
// @include        https://mail.google.com/*
// ==/UserScript==

window.addEventListener('load', function() {
if (unsafeWindow.gmonkey) {
 unsafeWindow.gmonkey.load('1.0', function(gmail) {
   function removeAdColumn() {
     if (gmail.getActiveViewType() == 'cv') {
    var sponsored = gmail.getConvRhsElement().lastChild;
    var sponsoredColumn = sponsored.parentNode.parentNode;
    sponsoredColumn.style.display = 'none';
    var convTable = gmail.getConvRhsElement().parentNode.parentNode.parentNode;
    convTable.width = '100%';
     }
   }
   gmail.registerViewChangeCallback(removeAdColumn);
   removeAdColumn();
 });
}
}, true);

When I get some more time I'll have to explore the power of this new API - and write a proper full width script that moves the buttons to a different location!

UML is dead, long live LSD

As part of a project hand-over I've recently done at a client there was the inevitable set of "standard" documents to write, such as a Design Document and a Troubleshooting guide.

There was no standard template for the Design Document, so amongst other things I included some UML class diagrams.


I was not specifically asked to include UML diagrams, but there was no objection to including them.

On reflection I haven't been asked by a client to specifically do UML diagrams for about 2 years now. Is this an indicator that UML is dead?

If UML is on its last legs what has replaced it? Seems to be what we had before Booch, Rumbaugh and the whole UML craze came along: ah-hoc component diagrams and sequence diagrams. (Although component and sequence diagrams are part of UML the UML versions have a particular look and convention to them.) Specs also seem to be getting thinner. I think the mainstreaming of Agile and Lean Software Development (LSD) ideas are taking effect.

btw, if you are using Eclipse there are a bunch of UML plugins that can help you draw UML diagrams. Most are a real pain to install. AmaterasUML, however, is very easy to install and has just enough features to make it useful (e.g. drag a Java class from the Java view and just drop it on to an AmaterasUML diagram view).

Minimial rsync/ssh for Windows

Every now and again I'm required to synchronize files between Windows and UNIX, usually a Windows desktop and UNIX server. Some of the commons tools for this task include a Samba Samba / robocopy combination, or installing the UNIX-like cygwin tools to use rsync/scp.

I personally think for this purpose cygwin is overkill and robocopy is slow and not very intuitive to use. With this bias in mind I was happy to stumble upon this article on installing a minimal rsync/ssh package on Windows. I must try this package out next opportunity!

RedHat buys then open sources AMQP implementation

How did I miss this event?

Previously I've blogged about Advanced Message Queue Protocol (AMQP), an effort by JPMorgan to commoditize messaging and break the back (i.e. save on the hugely expensive licensing costs) of the messaging behemoths Tibco and IBM.

According to this article RedHat bought an AMQP implementation from JPMorgan, improved it, then gave the implementation to the Apache Foundation as the QPid project. Currently there is a Java and C++ broker with clients in C++, Java (JMS), Ruby, Pyhton and C# for .NET.

Red Hat has the ambitious goal of enabling a server to process 1 million messages per second.

I guess JP didn't have an interest in maintaining the AMQP implementation, but RedHat are not exactly cheap when it comes to maintenance fees (but so far cheaper than Tibco and IBM).

Update: Microsoft have joined the AMQP working group!

When Network Cables Attack

I bet you this is Time-Warner's data center...

(Picture from http://kevinremde.members.winisp.net/images/yellow_wall.jpg)

Swing gets even slower by using Vector Graphics

In previous posts I've lamented how Swing had great potential to capture the hearts and minds of GUI developers, but stubbornness on not using native widgets and a bad implementation means they have forfeited GUI applications to competitors like .NET. Most new thick GUIs that our financial clients ask us to develop are .NET GUIs (C# is the language of choice).

I was reminded recently of the pitfalls of the Swing draw-each-widget-pixel-by-pixel approach when I used a simple GUI like JConsole to access an application that I was JMX-ising:


Horrible refresh problem. Swing applications seem to be much more prone to this than native applications.

Can I blame this on the pixel-by-pixel approach, or just Swing's implementation of it? One other implementation of pixel-by-pixel that I played with a few years ago is Thinlet, a remarkably tiny GUI toolkit. I have never put a Thinlet application into production (only recently have clients gotten comfortable with the LGPL license), but I was impressed with the speed of it.

Recently I read an article Nimbus, a new sexy look-and-feel that the Sun Swing engineers are working hard on. When I read that Nimbus used vector graphics I cringed. Even with faster CPUs I bet the performance will still be similar to the current pixel-by-pixel approach. i.e. sucks

Don't get me wrong. Having a vector graphics option is good for some GUIs, but I really wish Sun (or whoever ends up driving the GUI side of Java) would also standardize a fast GUI toolkit. SWT is one option. Thinlet is another.

It is also interesting to note that Android, Google's Java-based framework for developing mobile phone applications, does not include any of the standard AWT or Swing libraries, instead using OpenGL C/C++ bindings.

Shame Sun, Shame for missing the opportunity to take over the desktop.

Dynamically add to the Eclipse JUnit classpath

When you run JUnit tests inside Eclipse something that may bite you is the fact that the compile classpath and runtime classpath are separate.

Let's say your Eclipse project directories are layed out something like the following:

PROJ_ROOT

• bin
• conf
• src
• test

where src contains your core application source code and test contains your JUnit tests.

When a JUnit test runs, by default the base run directory and the classpath root is the PROJ_ROOT directory. Therefore, if your unit test loads a file relative to the the PROJ_ROOT, say:

new FileInputStream("conf/context.xml");

or

Thread.currentThread().getContextClassLoader().getResourceAsStream("conf/context.xml");

it will work. If, however, you've told Eclipse that conf is a "src" directory (i.e. part of the compile classpath) and then try to load a file from somewhere in the classpath from your unit test like so:

Thread.currentThread().getContextClassLoader().getResourceAsStream("context.xml");

it will fail.

So what can you do to change the classpath for the unit tests that need it?

You can manually add the conf directory to the runtime classpath for just those unit tests that need them, but unfortunately so does everyone else in your team.

A cute trick is to change the thread classloader to a URLClassLoader, whose constructor allows you to add URLs (including directories) to the runtime classpath:

ClassLoader currentThreadClassLoader
 = Thread.currentThread().getContextClassLoader();

// Add the conf dir to the classpath
// Chain the current thread classloader
URLClassLoader urlClassLoader
 = new URLClassLoader(new URL[]{new File("conf").toURL()},
                      currentClassLoader);

// Replace the thread classloader - assumes
// you have permissions to do so
Thread.currentThread().setContextClassLoader(urlClassLoader);

// This should work now!
Thread.currentThread().getContextClassLoader().getResourceAsStream("context.xml");

To give credit where it is due the above is a slightly simplified version of a solution from my colleague Randy who came up with it when we worked together on a project earlier this year. His solution makes the assumption that the JVMs system classloader is a URLClassLoader (may not be true for all JVMs) and uses a bit of reflection magic to add to the classpath:

public void addURL(URL url) throws Exception {
  URLClassLoader classLoader
         = (URLClassLoader) ClassLoader.getSystemClassLoader();
  Class clazz= URLClassLoader.class;

  // Use reflection
  Method method= clazz.getDeclaredMethod("addURL", new Class[] { URL.class });
  method.setAccessible(true);
  method.invoke(classLoader, new Object[] { url });
}

addURL(new File("conf").toURL());

// This should work now!
Thread.currentThread().getContextClassLoader().getResourceAsStream("context.xml");

When Teachers Attack - Caught On Mobile Phone Camera

My own experience of High School showed me teachers were a mixed bag - some excellent, some very average and some very lazy. Occasionally good teachers had bad moments.

I've got mixed feelings about YouTube's 7 Scariest Teachers, a selection of teacher rants caught by students with mobile (cell) phone cameras. Should teachers, who are already struggling to teach unmotivated students, be exposed to public ridicule like this? On the other hand this could help weed out people who shouldn't be teaching.

What keywords do recruiters search for?

It's well known that recruiters / job agents do a lot of filtering of resumes using keyword searches. (I get a lot of spam from recruiters based on a technology I might have used once or twice in my career that I mentioned on my resume at some point in the past.)

But what keywords are hot right now? Marketing job website MktgLadder have made available a regularly updated Top 100 Keywords searched for by recruiters. I would like to see this feature added to some of the more IT-centric job sites like Dice.

Microsoft Silverlight is anti-competitive?

It seems that seven U.S. states consider Microsoft has been dragging its feet in providing information about its server protocols as required to do so when it lost an anti-trust case a few years ago. Fearing that technologies like Silverlight "substantially depend upon the browser" and that Internet Explorer is still bundled with Windows the states are asking for an extension to the anti-trust judgment.

Reading DOCX and other Office 2007 formats using older versions of Office

If you're like me and haven't had a need to upgrade from Office 2000 but get sent a file in one of the new Office 2007 formats then you might be able to open up these files in your old version of Office using the Microsoft Office Compatibility Pack, a free download.

There also seem to be a couple of free conversion services on the web if you're willing to upload your docs: docx2doc.com and docx-converter.com

I'll have to give OpenOffice a try some time :)

BEA Event Server based on open source Esper project

While reading Catching up with Esper: Event Stream Processing Framework I couldn't help but notice this little gem: BEA's recently launched WebLogic Event Server is based on a modified version of the open source Esper event stream processing (ESP) and complex event processing (CEP) project.

This is a departure from the normal behavior of BEA, which is buy a product/vendor if the other vendor is small enough or license it if the other vendor is big. I can only assume that there are not that many vendors in the CEP space and that Esper is reasonably mature.

So what value add does the BEA version contain? Not much info on this, but appears to offer an entire server architecture - including deployment, admin, security, etc - for those that don't want to build their own. Interestingly this product is not based on WebLogic, but is a more light-weight stack based around an OSGi model.

The Esper API looks suspiciously like the Apama API :)

The 4-Hour Workweek: Tips on managing your time in the era of email, IM, etc

Intro video on the Tim Ferriss book The 4-Hour Workweek.



Tim manages a medium-sized company with domestic and international customers. Like many of us in the digital age he has struggled to manage his time in an era with constant round-the-clock email/IM/cell phone/etc interruptions. Some of his tips:

• Batch email. Tim looks at his email only twice a day, 11am and 4pm. So that staff and customers won't get angry at not getting an immediate response Tim has an auto-responder that replied back telling people when he looks at his email, and if anything requires an urgent response then they should call him on his cell phone.
• 80/20 rule. If a customer takes up a lot of time but does not contribute much revenue then be minimally responsive to them. (I think it might also help to let the customer why they are not getting much of a response.)
• Outsource your life. Always have a plan for your day before getting out of bed. If something takes lots of time but can easily and cheaply be performed by others (e.g. research) then outsource it to them. Tim says that people in India and Canada are available for such work at $5 per hour. Check out websites like getfriday.com and workaholicsforhire.com
  

Banner Ad Blindness

This article presents research on how regular web surfers are used to ignoring ads, or even content that looks like ads or content that is in places where they expect ads to be. (Some nice heat maps of where people's eyes wandered on test web pages.)


How does the author propose to get more eyeballs on ads? Make the ads look like content :)

Sick Of Waiting For An Install, 75-Year-Old Woman Smashes Up Comcast Office With Hammer

After being on the receiving end of poor service from Time Warner I can really sympathize with this woman...

"Have I got your attention now?" asked Mona Shaw of the Comcast payment center employees as she smashed their keyboard, monitor and telephone...

"It's totally not like me to do stuff like this," said Shaw. "But it is so irresponsible and so disrespectful. I can't think of any company reacting that way. It's like they got you in their clutches and they'll do what they damn well please."

Something that might come in handy if you get the cable company runaround: contact info for cable company executives.

World's longest stack trace?

I really love all of the useful libraries that have grown out of the open source movement in the last couple of decades. It really makes assembling powerful applications out of freely available components very easy (as long as you use robust components!), but when the stack gets big and things go wrong it's kind of worrying when you see stack trace this long:

ERROR [org.jboss.ejb.plugins.LogInterceptor] EJBException in method: public abstract org.hyperic.util.pager.PageList org.hyperic.hq.bizapp.shared.MeasurementBoss.findMeasurementData(int,org.hyperic.hq.appdef.shared.AppdefEntityID,org.hyperic.hq.measurement.shared.MeasurementTemplateValue,long,long,long,boolean,org.hyperic.util.pager.PageControl) throws org.hyperic.hq.auth.shared.SessionNotFoundException,org.hyperic.hq.auth.shared.SessionTimeoutException,org.hyperic.hq.measurement.data.DataNotAvailableException,org.hyperic.hq.appdef.shared.AppdefEntityNotFoundException,org.hyperic.hq.authz.shared.PermissionException,org.hyperic.hq.measurement.MeasurementNotFoundException,java.rmi.RemoteException, causedBy:
java.lang.ArithmeticException: / by zero
 at org.hyperic.hq.measurement.server.session.DataManagerEJBImpl.getHistoricalData(DataManagerEJBImpl.java:800)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
 at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
 at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)
 at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)
 at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:54)
 at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
 at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:118)
 at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:141)
 at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)
 at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:300)
 at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
 at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
 at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
 at org.jboss.ejb.Container.invoke(Container.java:873)
 at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:415)
 at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)
 at $Proxy259.getHistoricalData(Unknown Source)
 at org.hyperic.hq.bizapp.server.session.MeasurementBossEJBImpl.findMeasurementData(MeasurementBossEJBImpl.java:1586)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
 at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
 at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:149)
 at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:154)
 at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:54)
 at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
 at org.hyperic.hq.application.HQApp$Snatcher.invokeNextBoth(HQApp.java:118)
 at org.hyperic.hq.application.HQApp$Snatcher.invokeNext(HQApp.java:141)
 at org.hyperic.txsnatch.TxSnatch.invoke(TxSnatch.java:71)
 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:106)
 at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
 at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
 at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:153)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
 at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
 at org.jboss.ejb.Container.invoke(Container.java:873)
 at sun.reflect.GeneratedMethodAccessor458.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
 at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:155)
 at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:104)
 at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:179)
 at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:165)
 at org.hyperic.hq.application.HQApp$Snatcher.invokeProxyNext(HQApp.java:135)
 at org.hyperic.txsnatch.ProxySnatch.invoke(ProxySnatch.java:37)
 at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
 at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
 at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:97)
 at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
 at $Proxy231.findMeasurementData(Unknown Source)
 at org.hyperic.hq.ui.action.resource.common.monitor.visibility.CurrentHealthAction.execute(CurrentHealthAction.java:123)
 at org.apache.struts.tiles.actions.TilesAction.execute(TilesAction.java:73)
 at org.hyperic.hq.ui.action.BaseRequestProcessor.processActionPerform(BaseRequestProcessor.java:63)
 at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
 at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
 at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672)
 at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:574)
 at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:499)
 at org.apache.struts.tiles.UrlController.execute(UrlController.java:89)
 at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:875)
 at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
 at org.apache.jsp.portal.ColumnsLayout_jsp._jspx_meth_tiles_insert_0(Unknown Source)
 at org.apache.jsp.portal.ColumnsLayout_jsp._jspx_meth_c_forEach_1(Unknown Source)
 at org.apache.jsp.portal.ColumnsLayout_jsp._jspx_meth_c_forEach_0(Unknown Source)
 at org.apache.jsp.portal.ColumnsLayout_jsp._jspService(Unknown Source)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672)
 at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:574)
 at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:499)
 at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:966)
 at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:604)
 at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)
 at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)
 at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)
 at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)
 at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
 at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_tiles_insert_3(Unknown Source)
 at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_html_html_0(Unknown Source)
 at org.apache.jsp.portal.MainLayout_jsp._jspService(Unknown Source)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672)
 at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463)
 at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398)
 at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
 at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
 at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
 at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:239)
 at org.apache.struts.tiles.TilesRequestProcessor.internalModuleRelativeForward(TilesRequestProcessor.java:341)
 at org.apache.struts.action.RequestProcessor.processForward(RequestProcessor.java:572)
 at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:221)
 at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
 at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672)
 at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463)
 at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398)
 at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
 at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
 at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
 at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:398)
 at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:318)
 at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:241)
 at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
 at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.hyperic.hq.ui.AuthenticationFilter.doFilter(AuthenticationFilter.java:110)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.hyperic.hibernate.filter.SessionFilter$1.run(SessionFilter.java:59)
 at org.hyperic.hq.hibernate.SessionManager.runInSessionInternal(SessionManager.java:78)
 at org.hyperic.hq.hibernate.SessionManager.runInSession(SessionManager.java:68)
 at org.hyperic.hibernate.filter.SessionFilter.doFilter(SessionFilter.java:57)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.hyperic.hq.product.servlet.filter.JMXFilter.doFilter(JMXFilter.java:324)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
 at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:159)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
 at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
 at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
 at java.lang.Thread.run(Unknown Source)

(Stack trace from the pretty good Hyperic HQ monitoring tool)

The 8-bit tie available now!

Nostalgic for the video games of old on the Atari ST, Commodore 64, ZX Spectrum, etc? Order your 8-bit tie now!

Browser detection using known HTML parser bugs

If you've developed webapps then it is likely you've had to detect what browser is accessing the site and then use javascript or CSS to do something special for that browser. This dude has found a way to use known HTML parser rendering bugs to identify Firefox, IE, Safari, Opera, Lynx and other browsers using just plain HTML. Here is some sample code (I must get round to converting this to Java):

#!/usr/bin/perl
print qq{
<img /src\x00="ie.gif"
  /''src\x00="firefox1_5.gif"
 /''src="firefox2_0.gif"
 /""src="gecko_others.gif"
 "s\x00rc="safari2.gif"
 "src="safari3.gif"
 ""src="konqueror.gif"
 src\x00="w3m.gif"
 src\x0c="opera.gif"
 src="others.gif"
 src="lynx.gif"
/> };

This technique is reported to work with many HTML tags, not just IMG. Clever hack!

More human-friendly Java GC timestamps

Update: If you are using the Sun (Oracle) JVM version 6u4 or newer then using the command line option -XX:+PrintGCDateStamps instead of -XX:+PrintGCTimeStamps will output a current date rather than a relative time in the GC logs.

If you've working on a medium to large Java project then one of the aspects of application performance you're probably (should be!) monitoring is garbage collection.

If you're using the Sun JVM or compatible JVM then I recommend you enable verbose garbage collection output with at least the following command-line options:

java -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc ...

This will print something like the following to stdout :

83.304: [GC 83.304: [DefNew: 890K->64K(960K), 0.0044947 secs] 890K->217K(5056K), 0.0046062 secs]
83.390: [GC 83.390: [DefNew: 960K->64K(960K), 0.0043936 secs] 1113K->376K(5056K), 0.0044796 secs]
83.534: [GC 83.534: [DefNew: 951K->64K(960K), 0.0046934 secs] 1263K->467K(5056K), 0.0048648 secs]
83.638: [GC 83.638: [DefNew: 960K->64K(960K), 0.0041580 secs] 1363K->658K(5056K), 0.0042530 secs]
83.783: [GC 83.784: [DefNew: 960K->64K(960K), 0.0030418 secs] 1554K->748K(5056K), 0.0031353 secs]
83.921: [GC 83.921: [DefNew: 960K->55K(960K), 0.0019457 secs] 1644K->794K(5056K), 0.0020575 secs]
91.655: [GC 91.655: [DefNew: 951K->19K(960K), 0.0012843 secs] 1690K->808K(5056K), 0.0013694 secs]
94.198: [Full GC (System) 94.198: [Tenured: 788K->759K(4096K), 0.0428238 secs] 883K->759K(5056K), [Perm : 2095K->2095K(12288K)], 0.0429641 secs]

As an alternative to writing to stdout - e.g. gc logging may get mixed up with a lot of non-gc related output - you can have the JVM log gc to a file with timestamps automatically enabled by using the -Xloggc option:

java -XX:+PrintGCDetails -Xloggc:logs/gc.log ...

One small annoyance I have with this logging is that the timestamp - which is of the format SSSS.MMMM where SSSS it the elapsed seconds and MMMM is the elapsed millisecs - is relative to when the JVM started. I would find it more useful if the timestamp told me the date and time of day that the logging occurred.

So here is a short list of requirements for more "human-friendly" timestamping:

• Log a timestamp that contains the date and time of day of the logging event, preferably in a configurable format. e.g. see java.text.SimpleDateFormat
• Work in a JVM-neutral and platform-neutral way

Given the above requirements what are my options?

Diving into the openjdk source code I found that the verbose gc logging is implemented in native C code.

From Java 5 onwards some of the garbage collection information is exposed via the GarbageCollectorMXBean JMX bean, but what is not exposed is any type of notification/callback when the gc actually occurs or more collector-specific information. e.g. for the parallel garbage collector how much time was spent in stop-the-world phase and how much time was spent in other phases.

Given this info I found two solutions:

1. Use a native wrapper around the JVM that timestamps messages logged to stdout

For this option I could use something like the very mature and useful utility Java Service Wrapper that provides this sort of wrapper around stdout (among other useful features), or possibly roll my own native code that redirects stdout (nightmare!).

A disadvantage of this approach is that you will have to deploy native libraries along with your application, which may not be easy to integrate into application servers, clustered environments, etc.

2. Create a thread that occasionally polls the GarbageCollectorMXBeans and logs the number of collections and collection time since the last poll

The code for this approach looks something like this:

package robertmaldon.gc;

import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;

public class GCTimeStamper implements Runnable {

  private SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS");

  private GarbageCollectorMXBean minorGCBean;
  private long lastMinorCollectionCount;
  private long lastMinorCollectionTime;

  private GarbageCollectorMXBean fullGCBean;
  private long lastFullCollectionCount;
  private long lastFullCollectionTime;

  public GCTimeStamper() {
  }

  public void run() {
      List<GarbageCollectorMXBean> gcMBeans = ManagementFactory.getGarbageCollectorMXBeans();

      for (GarbageCollectorMXBean gcBean : gcMBeans) {
          if ("Copy".equals(gcBean.getName())) {
              minorGCBean = gcBean;
          } else if ("MarkSweepCompact".equals(gcBean.getName())) {
              fullGCBean = gcBean;
          } else if ("ParNew".equals(gcBean.getName())) {
              minorGCBean = gcBean;
          } else if ("ConcurrentMarkSweep".equals(gcBean.getName())) {
              fullGCBean = gcBean;
          } else {
              System.err.println("Unable to classify GarbageCollectorMXBean [" + gcBean.getName() + "]");
          }
    }

    try {
        while (true) {
            if (minorGCBean.getCollectionCount() != lastMinorCollectionCount) {
                long diffCount = minorGCBean.getCollectionCount() - lastMinorCollectionCount;
                long diffTime = minorGCBean.getCollectionTime() - lastMinorCollectionTime;
                System.out.println(getTimeStamp() + "Minor GC x " + diffCount + ", " + diffTime + " millisecs");
    
                lastMinorCollectionCount = minorGCBean.getCollectionCount();
                lastMinorCollectionTime = minorGCBean.getCollectionTime();
            }

            if (fullGCBean.getCollectionCount() != lastFullCollectionCount) {
                long diffCount = fullGCBean.getCollectionCount() - lastFullCollectionCount;
                long diffTime = fullGCBean.getCollectionTime() - lastFullCollectionTime;
                System.out.println(getTimeStamp() + "Full GC x " + diffCount + ", " + diffTime + " millisecs");
    
                lastFullCollectionCount = fullGCBean.getCollectionCount();
                lastFullCollectionTime = fullGCBean.getCollectionTime();
            }

            // Sleep a little bit before the next poll
            Thread.sleep(1000);
          }
        } catch (Exception ex) {
            // Do nothing except exit this thread
        }
    }

  private String getTimeStamp() {
      Date now = new Date();

      long upTime = ManagementFactory.getRuntimeMXBean().getUptime();
      long upTimeSecs = upTime / 1000;
      long upTimeMillis = upTime % 1000;

      return dateFormat.format(now) + " [" + upTimeSecs + "." + upTimeMillis + "] ";
  }
}

The above class would be started with a new Thread(new GCTimeStamper()).start() or a TaskExecutor, and you can replace the System.out statements with calls to log4j or whatever.

The output of this class mixed with the gc output looks something like this:

83.304: [GC 83.304: [DefNew: 890K->64K(960K), 0.0044947 secs] 890K->217K(5056K), 0.0046062 secs]
83.390: [GC 83.390: [DefNew: 960K->64K(960K), 0.0043936 secs] 1113K->376K(5056K), 0.0044796 secs]
83.534: [GC 83.534: [DefNew: 951K->64K(960K), 0.0046934 secs] 1263K->467K(5056K), 0.0048648 secs]
83.638: [GC 83.638: [DefNew: 960K->64K(960K), 0.0041580 secs] 1363K->658K(5056K), 0.0042530 secs]
83.783: [GC 83.784: [DefNew: 960K->64K(960K), 0.0030418 secs] 1554K->748K(5056K), 0.0031353 secs]
83.921: [GC 83.921: [DefNew: 960K->55K(960K), 0.0019457 secs] 1644K->794K(5056K), 0.0020575 secs]
2007-09-28 17:03:59,037 [84.80] Minor GC x 6, 22 millisecs
91.655: [GC 91.655: [DefNew: 951K->19K(960K), 0.0012843 secs] 1690K->808K(5056K), 0.0013694 secs]
2007-09-28 17:04:07,036 [92.79] Minor GC x 1, 1 millisecs
94.198: [Full GC (System) 94.198: [Tenured: 788K->759K(4096K), 0.0428238 secs] 883K->759K(5056K), [Perm : 2095K->2095K(12288K)], 0.0429641 secs]
2007-09-28 17:04:10,036 [95.79] Full GC x 1, 42 millisecs

This class will print the approximate date/time a gc occurred as well as the time relative to the start of the JVM, so you can then use that to correlate with the more verbose output in the gc.log file.

Another interesting problem is determining which gc was minor and which was full (major). If you're using the Sun JVM and the default gc settings then a collector named "Copy" does minor collections and a collector named "MarkSweepCompact" does full collections. If you're using the Sun JVM with the -XX:+UseConcMarkSweepGC option then a collector named "ParNew" seems to do minor collections and a collector named "ConcurrentMarkSweep" seems to do full collections. (I say "seems to" because the behavior changes depending on if you're running on a one cpu or multi-cpu machine.)

Not an ideal solution, but if anybody knows a better way I would be interested in hearing from you!

The most complete list of -XX options for Java 6 JVM

The Sun JVM has many, many, many non-standard options (i.e. may not be supported by other JVMs), and most are not documented on the official Sun website.

Here is a pretty comprehensive listing of the -XX options for the Java 6 JVM.

Google, Yahoo Sued For Stealing Names From Tanzanian Tribes

Another frivolous lawsuit?

...He claims that both Google and Yahoo stole their names from Tanzanian tribes -- and now they should pay up. Specifically, he claims that Google took its name from the Gogo tribe and Yahoo took its name from the Yao tribe. Conveniently, this guy happens to be a descendant of both tribes. He's merely asking for both companies to pay $10,000 each to every member of both tribes, going back three generations...

How do you say "MSN", "Ask.com" or "Slashdot" in Tanzanian?

Bush Cheney '08

A trend that appears to be gaining momentum on both sides of politics...

• www.bushcheney08.org Petition To Amend the Constitution
  
• RightWingSwag bumper sticker
• Stay the Course t-shirts

Update: US election hots up: Palin in a bikini, Battlestar Galactica stars enter the race

How to Launder Money

The clearest explanation of money laundering that I've ever seen :)

(Picture from this blog post)

Programmatically Configuring log4j - and the Future of Java Logging

log4j is probably the most common logging framework in use in the Java world. Even if a project/framework uses commons-logging most deployments configure log4j to be the underlying logger.

log4j is usually configured via a properties file (most common, in my experience) or an XML file (more powerful). There are a few situations, however, where I've found it useful to programmatically configure log4j, and since there is very little info on the web on how to do this I thought I'd write a short blog entry on the subject.

Why configure log4j programmatically?

Two situations I've found it useful to configure log4j in code are:

• During running of Unit Tests. Since unit tests are fine grained it helps to enable more verbose logging for just the area a particular test targets instead of for the whole suite. Also, if you are running single test inside your IDE you may want to output to the console, but if you run your whole suite from the command line you may want to output to a file.
  
• Changing logging dynamically via JMX. log4j has some basic JMX support, but if you want to do anything mildly complicated then you'll have to roll your own.

log4j initialization

Whenever a Logger is first used log4j goes through a default initialization procedure to find a log4j.properties or a log4j.xml configuration file. If the initialization procedure fails you typically get an error message like this:

log4j:WARN No appenders could be found for logger (net.sf.ehcache.CacheManager).
log4j:WARN Please initialize the log4j system properly.

when a Logger is used.

The simplest way to detect if log4j was initialized or not is to ask the root logger if it has any appenders attached to it:

import org.apache.log4j.Logger;

Logger root = Logger.getRootLogger();
if (!root.getAllAppenders().hasMoreElements()) {
// No appenders means log4j is not initialized!
}

A minimal default log4j configuration might be to add a single console appender to the root logger; this is in fact what the convenience method BasicConfigurator.configure() does:

import org.apache.log4j.ConsoleAppender;
import org.apache.log4j.Logger;
import org.apache.log4j.PatternLayout;

Logger root = Logger.getRootLogger();
root.addAppender(new ConsoleAppender(
    new PatternLayout(PatternLayout.TTCC_CONVERSION_PATTERN)));

If not explicitly set the default the log level will be DEBUG.

Example

So let's put these things together in a scenario: If I run my unit test suite from Ant then I add a log4j.properties to the classpath that directs log4j logging to a file. If I run a unit test from inside Eclipse then I deliberately want log4j initialization to fail, allowing me to initialize it programatically; log everything from the robertmaldon.moneymachine package to the console at DEBUG level, and log everything else to the console at INFO level.

package robertmaldon.moneymachine;

import org.junit.BeforeClass;

import org.apache.log4j.ConsoleAppender;
import org.apache.log4j.Logger;
import org.apache.log4j.PatternLayout;

public class OrderTest {

@BeforeClass
public static void classSetUp() {
  Logger rootLogger = Logger.getRootLogger();
  if (!rootLogger.getAllAppenders().hasMoreElements()) {
      rootLogger.setLevel(Level.INFO);
      rootLogger.addAppender(new ConsoleAppender(
             new PatternLayout("%-5p [%t]: %m%n")));

      // The TTCC_CONVERSION_PATTERN contains more info than
      // the pattern we used for the root logger
      Logger pkgLogger = rootLogger.getLoggerRepository().getLogger("robertmaldon.moneymachine");
      pkgLogger.setLevel(Level.DEBUG);
      pkgLogger.addAppender(new ConsoleAppender(
             new PatternLayout(PatternLayout.TTCC_CONVERSION_PATTERN)));
  }
}
}

The above code creates a new Logger and chains it to the root logger. Instances of Logger can not be instantiated directly, but have to be created from a LoggerRepository (a factory pattern).

The equivalent log4j.properties would look something like:

log4j.rootLogger=INFO, A

log4j.appender.A=org.apache.log4j.ConsoleAppender log4j.appender.A.layout=org.apache.log4j.PatternLayout
log4j.appender.A.layout.ConversionPattern=%-5p [%t]: %m%n

log4j.logger.robertmaldon.moneymachine=DEBUG, B

log4j.appender.B=org.apache.log4j.ConsoleAppender log4j.appender.B.layout=org.apache.log4j.PatternLayout
log4j.appender.B.layout.ConversionPattern=%r [%t] %p %c %x - %m%n

So not too difficult to do after all :)

The Future of Java Logging

In recent weeks I've noticed some discussion and dissension in the Java logging world.

Some implementation issues with commons-logging spurred a number of the Apache projects to switch to a replacement called SLF4j (Simple Logging Framework).

The original creator of log4j has been hard at work on a successor called LOGBack, which is an evolution of log4j rather than a revolution. e.g. smaller memory footprint, faster, better filtering and evaluation capabilities.

One feature of LOGBack that really caught my eye was that of parameterized logging. In log4j you might write logging code like:

if( logger.isDebugEnabled() ) {
  logger.debug( "User with account "
      + user.getAccount() + " failed authentication; "
      + "supplied crypted password " + user.crypt(password)
      + " does not match." );
}

The isDebugEnabled() check is usually a performance thing - there is no need to call expensive methods like the crypt() method above if the current logging level is higher than DEBUG. The equivalent code in LOGBack would be:

logger.debug( "User with account {} failed authentication; "
            + "supplied crypted password {} does not match.",
            user.getAccount(), user.crypt(password) );

which allows LOGBack to defer calling getAccount() and crypt() until it has determined that the log level will cause the logging to be generated. The less "if"s the better!

LOGBack natively implements the SLF4j APIs, but provides a bridge for applications currently using the log4j APIs. SLF4j provides a bridge for applications currently using the commons-logging APIs.