Saturday, June 23, 2007

If Architects Had To Work Like Web Designers

A comical comparison of design/engineering in the physical world versus the expectations of unlimited flexibility and quick delivery of design/engineering in the computer world:

Dear Mr. Architect:

Please design and build me a house. I am not quite sure of what I need, so you should use your discretion. My house should have somewhere between two and forty-five bedrooms. Just make sure the plans are such that the bedrooms can be easily added or deleted...

Please take care that modern design practices and the latest materials are used in construction of the house, as I want it to be a showplace for the most up-to-date ideas and methods. Be alerted, however, that kitchen should be designed to accommodate, among other things, my 1952 Gibson refrigerator...

Also, do not worry at this time about acquiring the resources to build the house itself. Your first priority is to develop detailed plans and specifications. Once I approve these plans, however, I would expect the house to be under roof within 48 hours.

Friday, June 22, 2007

Configuring WebSphere to keep the Java code generated from a JSP

One of the more useful techniques you can use to debug the cause of a JSP compilation failure is to view the Java source code that the JSP engine generates from a JSP. You would think that WebSphere (Web's Fear as my colleague Solomon calls it) would make it easy to get the generated Java source - e.g. a checkbox in the WebSphere admin console - but you would be wrong.

Configuring WebSphere to leave JSP generated Java source is done on a per-application basis. It is switched on by modifying a parameter in the WebSphere-specific WEB-INF/ibm-web-ext.xmi deployment descriptor. This file is generated automatically by tools such as WSAD (aptly pronounced wah-sad) or by WebSphere when it installs an application.

The default WEB-INF/ibm-web.ext.xmi looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<webappext:WebAppExtension xmi:version="2.0" xmlns:xmi=
xmlns:webappext="webappext.xmi" xmlns:webapplication="webapplication.xmi" xmi:id="WebAppExtension_1"
<webApp href="WEB-INF/web.xml#MyWebApp"/>
<jspAttributes xmi:id="JSPAttribute_1" name="useThreadTagPool" value="true"/>
<jspAttributes xmi:id="JSPAttribute_2" name="verbose" value="false"/>
<jspAttributes xmi:id="JSPAttribute_3" name="deprecation" value="false"/>
<jspAttributes xmi:id="JSPAttribute_4" name="reloadEnabled" value="true"/>
<jspAttributes xmi:id="JSPAttribute_5" name="reloadInterval" value="5"/>
<jspAttributes xmi:id="JSPAttribute_6" name="keepgenerated" value="true"/>
<!-- <jspAttributes xmi:id="JSPAttribute_7" name="trackDependencies" value="true"/> -->
<jspAttributes xmi:id="JSPAttribute_8" name="jspCompileClasspath" value=""/>

To force WebSphere to leave the generated Java source you need to change the value of the keepgenerated attribute to true and restart your application. WebSphere should then leave the generated source in the "scratch dir" found under profile_root/temp.

More WebSphere JSP engine configuration parameters here.

New Development Methodology Acronyms

A slightly cynical list of acronyms describing today's most common development methodologies:

  • Cover Your Ass Engineering (CYAE) - The driving force behind most individual efforts is to make sure than when the shit hits the fan, they are not to blame
  • Development By Denial (DBD) - Everybody pretends there is a method for what’s being done
  • Get Me Promoted Methodology (GMPM) - People write code and design things to increase their visibility
  • Not My Problem (NMP) - All complex, complicated, expensive, or otherwise troublesome decisions/features/issues are pushed into someone else’s module
  • Not Allowed to Do Development (NADD) - No actual development is tolerated by the development team. Everyone with any technical knowledge at all is at least 3 levels too low on the corporate ladder to be allowed to make even trivial decisions
  • Shovel-Driven Development (SDD) - Get it out the door as quickly as possible
  • Blog Driven Development (BDD)- Developers who are constantly thinking about the subject of their next blog post. Nearly every somewhat interesting line of code they write is extracted into a blog post
  • Copy/Paste/Modify (CPM)
  • Not Invented Here (NIH)
  • Budget Driven Development (BDD)
  • Never Ending Story Development (NESD)
  • I Wish I Was Somewhere Else (IWIWSE) - Produces some of the most unmotivated code in existence
  • Decapitated Chicken Process (DCP) - A time honored micromanagement technique where each day managers identify a drastic emergency and require developers drop what they are doing
  • It’s What They Wanted Development (IWTWD) - Absolving oneself of all accountability by inventing a group of people known as “they” and blaming them for one’s own inability to design and develop a usable system
  • Visibility Driven Development (VDD) - We’re selling the company, so the more times the not-really-ever-going-to-be-product squeaks, whistles, spins, churns, flashes, or wobbles, the better
  • Operation Death Star (ODS) - Develop until one critical function is operational, declare the product “fully operational” and schedule a test. Watch all the important people jump ship just before the test. You’ll feel it in the force when the test blows something up.
  • Blame QA (BQA)
  • Ping Pong Development (PPD) - A development methodology where you enlist a minimum of two stakeholders with mutually exclusive requirements and visions and then have them directly harass the developer constantly
  • Worry About It Later Driven Development (WAILDD)
  • Must Use Specific Technology Development (MUSTD)
  • Get The Money, Get The @!# Money! (GTMGTFM)

Thursday, June 14, 2007

Web Privacy : Spotting Web Bugs

What is a Web Bug?

From the Web Bug FAQ:

A Web Bug is a graphics on a Web page or in an Email message that is designed to monitor who is reading the Web page or Email message. Web Bugs are often invisible because they are typically only 1-by-1 pixel in size. They are represented as HTML IMG tags. For example, here are two Web Bugs recently found on Quicken's home page (

<img style="display: none;" src="" border="0" height="1" width="1" />

<img src="" border="0" height="1" width="1" />

The two Web Bugs were placed on the home page by Quicken to provide "hit" information about visitors to DoubleClick and MatchLogic (AKA,, two Internet advertising companies.

Information that a Web Bug can track includes:
  • The IP address of the computer that fetched the Web Bug
  • The URL of the page that the Web Bug is located on
  • The URL of the Web Bug image
  • The time the Web Bug was viewed
  • The type of browser that fetched the Web Bug image
  • A previously set cookie value
The 1x1 IMG bug is still very common, probably because the technique works across all browsers and even when javascript is disabled. However, a Web Bug could be a SCRIPT (javascript), IFRAME, APPLET or some other HTML element.

Are Web Bugs bad?

That is up to you to decide :) Many web sites rely on advertising, so having stats on who uses their sites and how the sites are used provides vital marketing information.

On the other hand information privacy is a hot topic with recent criticism over Google's lack of privacy policies and how Microsoft is developing algorithms that correctly guess the gender and age of web surfers.

In the interest of full disclosure my blog has a Google Analytics Web Bug that I use to review which of my posts are the most popular.

How Can I spot a Web Bug?

You can view the HTML source of a web site and look for images with a 1x1 pixel size. Doing this inspection manually is pretty impractical for large web sites.

However, I came across this genius Greasemonkey script that, after a web page has loaded, will search for any 1x1 images and change their size to 101x101 and color to bright green! The result is that when you are scrolling through a web page any Web Bugs in the page will be obvious.

(If you're using Firefox I highly recommend you install the Greasemonkey addon. It allows you to attach user defined javascript to any web page, giving you the opportunity to change the look or behavior of a web site to just the way you like it.)

That script has a couple of limitations:
  1. It highlights all 1x1 images as possible Web Bugs. There are a lot of sites that still use spacer GIFs to control formatting in older browsers.
  2. It does not highlight non-IMG Web Bugs.
To get around the limitations I extended the script to do the following:
  1. Highlight 1x1 images only if the URL to the image contains query parameters. e.g.
  2. For non-IMG Web Bugs list the URLs in bright green at the bottom of the page. I constructed a list of some of the worst offending Web Bug sites from the SecuritySpace Web Bug Tracking Survey.
So here is the modified script:

// ==UserScript==
// @name Detect Web Bugs
// @namespace
// @description Attempt to detect web bugs by expanding the size of bugged 1x1 images (based on original idea and listing other possible bugs at the bottom of the web page.
// ==/UserScript==

// ver 1.0 @ 2007-06-13
// First release

( function(){

function logWebBug(tagName, text) {
var bugText = document.createTextNode("WeB BuG [" + tagName + "] " + text);

var bugDiv = document.createElement("div");
bugDiv.border = "11"; = '#ff0000'; = '#00ff00';


window.addEventListener("load", function(e) {

// Search for 1x1 images that also contain a query string
var imgList = document.getElementsByTagName("img");
for (var i = 0; i < imgList.length; i++) {
if (imgList[i].src != "" && imgList[i].src.match(/\?/) &&
imgList[i].naturalWidth == 1 && imgList [i].naturalHeight == 1) {
imgList[i].width = "101";
imgList[i].height = "101";
imgList[i].alt = "WeB BuG";
imgList[i].border = "11";
imgList[i].style.borderColor = '#ff0000';
imgList[i].style.backgroundColor = '#00ff00';

logWebBug("img", imgList[i].src);

// Search for other types of web bugs
var tags = ['img', 'script', 'iframe', 'style', 'embed', 'applet', 'object'];
var domains = ['', '', '', '',
'', '', '', '',
'', '', '', '',
'', '', '', '',
'', '', '',
'', '', '',
'', '', '',
'', '', '', '',
'', '', '',
'', '', '', '',
'', '', '',
'', '', '',
'', '', ''];
for (var i = 0; i < tags.length; i++) {
var elementList = document.getElementsByTagName(tags[i]);
for (var j = 0; j < elementList.length; j++) {
for (var k = 0; k < domains.length; k++) {
if (elementList[j].src.match(domains[k])) {
logWebBug(tags[i], elementList[j].src);

}, false);


If you are running Firefox you can install the script by doing the following:
  1. Install Greasemonkey if you haven't done so already
  2. Manually install the script above or install by clicking on this download link
  3. Surf away!
It shouldn't take long to find Web Bugs - they're everywhere!. Below is an example from the bottom of the New York Times home page which contains a classic 1x1 Web Bug and lots of DoubleClick Web Bugs.

I've Found A Web Bug I Don't Like. How Do I Block It?

If you are using Firefox then install the Adblock Plus addon and add the URL of the Web Bug to the block list.

Saturday, June 09, 2007

Caught In The Act By Google Street View

Google Maps have recently added a "street view" feature where, if available, they will show you a picture of the location you've zoomed in on. (Nice of Google to catch up with Hopstop on this feature :))
Break and Enter?

The availability of this feature has started a new craze of searching for people doing the weirdest things, such as the dude above breaking and entering, a guy entering an adult book store or the dude below who is trying to rip off my blog :)

Does that sign say 'Will Code For Food'?

Geek Tech: The Wooden Computer

Ooooh, I bet many an office exec would like one of these.

If you can't wait for this mass market item then try this dude that does custom wood cases for laptops, desktops, monitors, disk drives, etc.

Monday, June 04, 2007

Adding a second docroot to a webapp

For some type of webapps it is common to have "static" resources (e.g. images, css style sheets, HTML files, PDF files) served up by Apache or IIS and "dynamic" resources (e.g. JSPs, ASP files) served up by a Java application server or .NET web engine.

Why split resource serving this way since Java/.NET can server up both static and dynamic resources? Sometimes for performance - if there are a lot of static resources a webserver written in C/C++ can serve them up faster than Java/.NET. Sometimes the application content is updated separately to the application logic - for example, in the media business the content of a newspaper or magazine website is updated several times a day, in the finance industry company research reports are updated daily.

In development it can be a real pain to run a static webserver as well as a Java/.NET application server. At a minimum running both + a IDE + other tools can bring your PC to its knees. As an experiment I decided to see how difficult it was to get a Java application server to serve resources from both it application context - or docroot - as well as relative from another location, or an external docroot.

Here is a rough specification of how I expected this stuff should work:
  • The webapp should first look for a resource relative to the directory configured through the user-defined Java property externalDocBase (e.g. -DexternalDocBase=Z:\pubroot\today);
  • If the resource could not be found relative to externalDocBase then look for the resource relative to the webapp context root.
externalDocBase on Windows may refer to a network share; on UNIX it may refer to an NFS mount.

For static resources this can be achieved fairly easily by creating a servlet to serve up static files something like this:

package robertmaldon.servlet;


import javax.servlet.ServletContext;

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpServletRequest;

public class ExternalDocBaseServlet extends HttpServlet {
private static org.apache.commons.logging.Log logger =

private static String externalDocBase;

static {
externalDocBase = System.getProperty("externalDocBase");
if (externalDocBase != null) {"externalDocBase set to [" + externalDocBase + "]");

* Create a File object relative to the external doc base.
private static File file(String name) {
return new File(externalDocBase, name);

public void doGet(HttpServletRequest request, HttpServletResponse response)
throws IOException {

ServletContext context = getServletContext();

File file = null;

// First see if the file can be served up from the external doc base
if (externalDocBase != null) {
file = file(request.getRequestURI());
if (file == null || !file.exists() || !file.canRead()) {
// Otherwise we look for the file relative to the application context
file = new File(context.getRealPath(request.getRequestURI()));

// Return a 404 if we could not find the file
if (!file.exists() || !file.canRead()) {

// Determine the file MIME type.
String mimeType = context.getMimeType(file.getAbsolutePath());
if (mimeType == null) {
context.log("Could not get the MIME type of [" + file.getAbsolutePath() + "]");

// Set content type

// Set content size

// Open the file and output streams
FileInputStream in = new FileInputStream(file);
OutputStream out = response.getOutputStream();

// Copy the contents of the file to the output stream in chunks
byte[] buf = new byte[1024];
int count = 0;
while ((count = >= 0) {
out.write(buf, 0, count);

and configuring this servlet to be the last servlet in the webapp (i.e. the default servlet) with the wild card mapping "/*" like this in web.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app xmlns=""
<!-- Last servlet in the webapp -->




This simple servlet (too simple?) can serve up any static resource, but what if you wanted to be able to serve up a dynamic resource like a JSP from an external docroot? Well, unfortunately this sort of behavior is application server specific.

For fun I dived into the source code for Tomcat 5.5.20 to discover how the JSP stuff works. Tomcat JSPs are compiled by a special servlet named org.apache.jasper.servlet.JspServlet, which is mapped to any file ending in .jsp. This servlet doesn't do much itself, delegating all of the hard work to helper classes. For the record here are the points in the Tomcat code that have to be modified to serve up a JSP relative to the externalDocBase :
  • org.apache.jasper.servlet.JspServlet.serviceJspFile(...) which check for the existence of the JSP file before delegating the work to helper classes;
  • org.apache.jasper.JspCompilationContext has a few "getResource" methods that locate or create a stream from the file.
Since we are diving into the Tomcat code, if you didn't want to define your own ExternalDocBaseServlet but instead wanted to modify the default Tomcat servlet org.apache.catalina.servlets.DefaultServlet then the place to add the externalDocBase logic is the org.apache.naming.resources.FileDirContext.file(String) method.

If you wanted to try this out for yourself then do the following:
  1. Download and unzip Tomcat 5.5.20
  2. Download my modifications to the Tomcat source code from here, unzip, and copy the compiled classes from externaldocbase-1.0\build to %TOMCAT_HOME%\common\classes
  3. Set the CATALINA_OPTS shell variable to point to your external doc root. e.g. set CATALINA_OPTS="-DexternalDocBase=Z:\pubroot\today"
  4. Run Tomcat. e.g. catalina.bat run

Sunday, June 03, 2007

Extending Spring PropertyPlaceholderConfigurer to consider the OS Platform

It is very common with Java to develop on Windows but deploy on UNIX. One of the main pains when switching between a Windows platform and a UNIX platform is the different ways they represent directory paths. e.g. the Windows path:


might have a UNIX equivalent that looks something like this:


The JVM actually does a pretty good job of coping with these path differences. For example, if you specify a path like "C:/software/hype" and the JVM knows it is running on Windows then it will internally convert the path to "C:\software\hype". It can even cope with a path containing mixed forward slashes and back slashes. e.g. "C:\software\hype/config/"

If you are Spring in your application then a common way to configure your application is to use the PropertyPlaceholderConfigurer class. This class replaces values in a Spring configuration file with property values from a properties file and/or system properties.

How do you manage the different paths when devloping for both Windows and UNIX? You could maintain two sets of property files - one for Windows devlopment and one for UNIX development - but this gets unwieldy when you have to define a lot of interconnected properties.

Instead I propose an alternative, a small extension to PropertyPlaceholderConfigurer that works like this:
  1. If you specify a property replacement like ${somevar} it will first look for a property named PLATFORM.somevar
  2. If it cannot find a property named PLATFORM.somevar then it will default to the old behaviour. i.e. look for a property named somevar
The value of PLATFORM is derived from the Java system property On Windows this property has the value of "Windows XP" or "Windows Me", etc. On Linux it has the value of "Linux". On Solaris it has the value of "SunOS". And so on. Using the actual value of can be a bit ugly (particularly when spaces are involved) so my extension maps to something more friendly. e.g. on Windows PLATFORM is mapped to "win"; on Solaris it is mapped to "sun".

So what does all of this look like together?

Let's say you define a simple POJO like this:

package robertmaldon.config;

public class PlatformHolder {
private String value;

public void setValue(String value) {
this.value = value;

public String toString() {
return "value=[" + value + "]";

Your Spring applicationContext.xml looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns=""

<bean class="robertmaldon.config.PlatformPropertyPlaceholderConfigurer">
<property name="locations">

<bean id="platformHolder" class="robertmaldon.config.PlatformHolder">
<property name="value" value="${somepath}"/>

Your file looks like this:


And finally a main class like this:

package robertmaldon.config;

import org.springframework.context.ApplicationContext;

public class PlatformMain {
public static void main(String[] args) throws InterruptedException {
ApplicationContext context = new ClassPathXmlApplicationContext("applicationContext.xml");

PlatformHolder holder = (PlatformHolder)context.getBean("platformHolder");

If running this progam on Windows the output of this progam would be:


and on any other platform the output would be:


So after all of that leadup here is what the PlatformPropertyPlaceholderConfigurer extension class looks like:

package robertmaldon.config;

import java.util.HashMap;
import java.util.Map;
import java.util.Properties;

import org.springframework.beans.factory.InitializingBean;
import org.springframework.beans.factory.config.PropertyPlaceholderConfigurer;

* <p>A small extension to org.springframework.beans.factory.config.PlatformPropertyPlaceholderConfigurer
* that, when asked to replace the placeholder "somevar", will first look for
* a property named "platformprefix.somevar", or failing a match will then look for
* the property "somevar".</p>
* <p>The "platformprefix" part of the placholder is derived from the Java system
* property "". For convenience the "" value is mapped to a
* prefix that is easier to type. For example, on Windows XP the value of ""
* is "Windows XP" and this class maps "platformprefix" to "win".</p>
* <p>This class has a default set of mappings (see DEFAULT_PLATFORM_PREFIX_MAPPINGS)
* which can be overridden by setting the property platformPrefixMappings.</p>
* <p>See for an extensive list of
* platform names used by the Java system property.</p>
public class PlatformPropertyPlaceholderConfigurer
extends PropertyPlaceholderConfigurer implements InitializingBean {


static {
DEFAULT_PLATFORM_PREFIX_MAPPINGS.put("Netware 4.11", "netware");
DEFAULT_PLATFORM_PREFIX_MAPPINGS.put("Windows 2000", "win");
DEFAULT_PLATFORM_PREFIX_MAPPINGS.put("Windows 2003", "win");
DEFAULT_PLATFORM_PREFIX_MAPPINGS.put("Windows NT (unknown)", "win");

private Map platformPrefixMappings;

private String platformPrefix;

* Override the default platform prefix mappings.
* @param platformPrefixMappings
public void setPlatformPrefixMappings(Map platformPrefixMappings) {
this.platformPrefixMappings = platformPrefixMappings;

* Attempt to determine the prefix to use for this platform.
* First check any user defined prefix mappings. If no match
* then check the default platform mappings.
public void afterPropertiesSet() throws Exception {
String platform = System.getProperty("");
if (platformPrefixMappings != null) {
platformPrefix = platformPrefixMappings.get(platform);
if (platformPrefix == null) {
platformPrefix = DEFAULT_PLATFORM_PREFIX_MAPPINGS.get(platform);

* Override the PropertyPlaceholderConfigurer.resolvePlaceholder(...) method
* to first look for a placeholder with the platform prefix.
protected String resolvePlaceholder(String placeholder, Properties props, int systemPropertiesMode) {
String propVal = null;
if (systemPropertiesMode == SYSTEM_PROPERTIES_MODE_OVERRIDE) {
if (platformPrefix != null) {
propVal = resolveSystemProperty(platformPrefix + "." + placeholder);
if (propVal == null) {
propVal = resolveSystemProperty(placeholder);
if (propVal == null) {
if (platformPrefix != null) {
propVal = resolvePlaceholder(platformPrefix + "." + placeholder, props);
if (propVal == null) {
propVal = resolvePlaceholder(placeholder, props);
if (propVal == null && systemPropertiesMode == SYSTEM_PROPERTIES_MODE_FALLBACK) {
if (platformPrefix != null) {
propVal = resolveSystemProperty(platformPrefix + "." + placeholder);
if (propVal == null) {
propVal = resolveSystemProperty(placeholder);
return propVal;