robertmaldon: November 2006

Back in the WebSphere 4.x days a light-weight performance monitoring infrastructure named, not surprisingly, Performance Monitoring Infrastructure (PMI) was introduced. The PMI infrastrcture basically consists of some application-upadted statistics "stubs" (i.e. handle objects containing counters or timers or something else) and a "collector" that periodically gathers data from the stubs. Before WebSphere 6.0 you had to use a thick client to view and analyze the PMI data, but starting with WebSpherer 6.0 the admin console generates pretty statistics graphs.

So you have a few different Spring-based applications deployed on WebSphere and after they go to production you think it would be a good idea to get gather some rough performance metrics on the applications (a pretty common scenario). What are your options?

Use a monitoring package that already has some Spring integration such as JAMon (see JamonPerformanceMonitorInterceptor) or JETM. These packages require a bit of work to integrate, requiring you to add servlets and/or JSPs and/or security to each of the applications you want to monitor.
Integrate the applications with PMI. Use the infrastructure already provided by WebSphere. No extra servlets/JSPs to roll out, no security changes.

PMI sounds like a cool option. Where do I find doco on it? Unfortunately this is very badly documented by IBM, but if you dig around the WebSphere InfoCenter you can find a few hints. But let me save you some trouble.

WebSphere uses PMI internally to monitor basic server health. IBM kindly provided "Custom PMI" hooks to allow developers to plug their applications into the PMI infrastructure. The way Custom PMI is supposed to work is as follows:

Each application needs an XML file which defines a number of stats "stubs" (see below for an example).
At application startup the application registers the stats stubs definition file with the PMI StatsFactory.
The application code must fetch and update the generated stats stubs at the appropriate places.

A stats stub definition file looks something like:


<?xml version="1.0"?>
<!DOCTYPE PerfModule SYSTEM "perf.dtd">

<PerfModule UID="com.ibm.websphere.pmi.jvmRuntimeModule">
<description>jvmRuntimeModule.desc</description>

<BoundedRangeStatistic name="jvmRuntimeModule.totalMemory" ID="1">
<participation>cross-family</participation>
<level>high</level>
<description>jvmRuntimeModule.totalMemory.desc</description>
<unit>unit.kbyte</unit>
<comment>Total memory in JVM runtime</comment>
<resettable>false</resettable>
<statisticSet>basic</statisticSet>
<zosAggregatable>false</zosAggregatable>
</BoundedRangeStatistic>

<CountStatistic name="jvmRuntimeModule.freeMemory" ID="2">
<participation>cross-family</participation>
<level>low</level>
<description>jvmRuntimeModule.freeMemory.desc</description>
<unit>unit.kbyte</unit>
<comment>Free memory in JVM runtime</comment>
<resettable>false</resettable>
<statisticSet>extended</statisticSet>
<zosAggregatable>false</zosAggregatable>
</CountStatistic>
...
</PerfModule>

Extract and have a look at com/ibm/websphere/pmi/xml/perf.dtd from $WEBSPHERE_INSTALL_DIR/lib/pmi.jar for the gory details.

WebSphere supports the following types of stats:

CountStatis tic (a simple counter)
AverageStatistic (take the average of some type of samples)
TimeStatistic (extends AverageStatistic, used to take the average of, say, time to execute a method)
BoundaryStatistic
BoundedRangeStatistic
DoubleStatistic
RangeStatistic

Of all of these types I can really only find a use for CountStatistic and AverageStatistic. If you know a good use for any of the others please let me know.

This sucks. The whole process of using the PMI API seems very static and very manual. Well, the PMI API was designed quite a few years ago, before more modern non-invasive techniques such as AOP were invented. Fortunately we can use a bit of Spring AOP magic to integrate your application with PMI without too much trouble.

ok, so after all of that preable what I really want to do is something like the following:

Use the Spring BeanNameAutoProxy to wrap PMI interceptors around all method calls (excluding bean setters) of beans defined in Spring.
Use the WebSphere admin console to enable/disable/log PMI data for my application.

Interestingly enough 1 is a little tricky. Spring AOP interceptors are usually completely unaware of which bean they are being called for, but in our case the method interceptor has to invoke the correct stats stub for the given bean. What we have to do, therefore, is write an autoproxy that glues interceptors to stats stubs. Fortunately BeanNameAutoProxy delegates most of its work to org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator, so we just extend AbstractAutoProxyCreator to create our own stats autoproxy named StatsAutoProxyCreator. (See the end of this post for the source code.)

Using StatsAutoProxyCreator the Spring config for our autoproxy looks something like this:


<bean name="mystats" class="robertmaldon.stats.StatsAutoProxyCreator">
<property name="beanNames"><value>MyPetStoreController</value></property>
<property name="statTypes"><value>time,count</value></property>
<property name="proxyTargetClass" value="true"/>
<property name="setName" value="basic"/>
</bean>

where

beanNames is a comma seperated list of beans to proxy
statTypes is a comma seperated list of stat types to collect. Currently supported values are "time" (average invocation time for a method) and "count" (number of times the method is invoked)
proxyTargetClass is a true/false to proxy the underlying class rather than interfaces implemented by the underlying class.
setName is a logical group name for the stats. If you do not specify setName it defaults to "basic". All "basic" stats are collected by default. If you use any other name then you will have to manually enable collection of the stats through the WebSphere admin console (Monitoring and Tuning -> Performance Monitoring Infrastructure (PMI) -> serverX -> Custom).

So with the stats autoproxy in place you start the application, log into the admin console, go to Monitoring and Tuning -> Performance Viewer -> Current Activity -> serverX -> Performance Modules , check the box next to MyPetStoreController (or whatever the bean name is), click on View Modules and watch the pretty graphs being generated.

You may also choose to save the PMI stats to a log file for later review.

Groovy stuff.

Appendix A : Special notes

If you are running WebSphere on a headless server then you will need to set the JVM command line option -Djava.awt.headless=true because the PMI console needs a display in which to draw graphs.
You will need to compile the code against pmi.jar and management.jar from the $WEBSPHERE_INSTALL_DIR/lib directory.

Appendix B : Code

StatsAutoProxyCreator.java


package robertmaldon.stats;

import java.util.Arrays;
import java.util.Iterator;
import java.util.List;

import org.springframework.aop.TargetSource;

import org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator;

/**
* NOTE: We had to reimplement BeanNameAutoProxyCreator instead of
* extending it because the list of bean names to proxy is declared as
* private.
*/
public class StatsAutoProxyCreator extends AbstractAutoProxyCreator {
private List beanNames;

private Integer[] statTypes;

private String setName;

public StatsAutoProxyCreator() {
super();
}

/**
* Set the names of the beans that should automatically get wrapped with
* proxies. A name can specify a prefix to match by ending with "*",
* e.g. "myBean,tx*" will match the bean named "myBean" and all beans
* whose name start with "tx".
*/
public void setBeanNames(String[] beanNames) {
this.beanNames = Arrays.asList(beanNames);
}

public void setStatTypes(String[] statTypes) {
List statTypesList = new java.util.ArrayList();
for (int i = 0; i < statTypes.length; ++i) {
   int statType = StatsConstants.getType(statTypes[i]);
   if (statType == 0) {
       throw new RuntimeException("Unknown stat type ["
                                  + statTypes[i] + "]");
   }

   statTypesList.add(new Integer(statType));
}

this.statTypes = (Integer[])statTypesList.toArray(new Integer[0]);
}

/**
* Give a name to the set of statistics created by this auto-proxy
* instance. The set name is used to enable and disable the set of
* stats as a whole.
*
* If the set name is not configured it defaults to "basic", which adds
* the statistics to the existing set of stats WebSphere PMI collects
* by default.
*/
public void setSetName(String setName) {
this.setName = setName;
}

/**
* Identify as bean to proxy if the bean name is in the configured list
* of names.
*/
protected Object[] getAdvicesAndAdvisorsForBean(Class beanClass,
String beanName, TargetSource targetSource) {
if (this.beanNames != null) {
   if (this.beanNames.contains(beanName)) {
       return StatsBuilder.getSingleton()
              .createTemplate(beanName, beanClass, statTypes, setName);
   }
   for (Iterator it = this.beanNames.iterator(); it.hasNext();) {
       String mappedName = (String) it.next();
       if (isMatch(beanName, mappedName)) {
           return StatsBuilder.getSingleton()
              .createTemplate(beanName, beanClass, statTypes, setName);
       }
   }
}
return DO_NOT_PROXY;
}

/**
* Return if the given bean name matches the mapped name.
* The default implementation checks for "xxx*" and "*xxx" matches.
* Can be overridden in subclasses.
* @param beanName the bean name to check
* @param mappedName the name in the configured list of names
* @return if the names match
*/
protected boolean isMatch(String beanName, String mappedName) {
return (mappedName.endsWith("*")
       && beanName.startsWith(
              mappedName.substring(0, mappedName.length() - 1)))
    ||
      (mappedName.startsWith("*")
           && beanName.endsWith(
                  mappedName.substring(1, mappedName.length())));
}
}

StatsBuilder.java


package robertmaldon.stats;

import java.lang.reflect.Method;

import java.util.List;
import java.util.Map;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import com.ibm.websphere.pmi.PmiDataInfo;
import com.ibm.websphere.pmi.PmiModuleConfig;

import com.ibm.wsspi.pmi.factory.StatisticActionListener;
import com.ibm.wsspi.pmi.factory.StatsFactory;
import com.ibm.wsspi.pmi.factory.StatsInstance;
import com.ibm.wsspi.pmi.factory.StatsTemplateLookup;

import com.ibm.wsspi.pmi.stat.SPIStatistic;

import com.ibm.wsspi.pmi.stat.SPICountStatistic;
import com.ibm.wsspi.pmi.stat.SPITimeStatistic;

/**
* This class generates the "custom" PMI glue code for a given Spring bean,
* including the MethodInterceptors.
*
* See StatsAutoProxyCreator for how the MethodInterceptors are applied to
* the beans.
*
* Design note: The underlying "custom" PMI factory is a singleton, so
* since this class is the main interface to the PMI factory it is also
* implemented as a singleton.
*/
public class StatsBuilder implements StatsTemplateLookup {
private Log logger = LogFactory.getLog(getClass());

private static StatsBuilder singleton = new StatsBuilder();

private Map templates = new java.util.HashMap();

private StatisticActionListener actionListener =
    new StatisticActionListenerImpl();

public static StatsBuilder getSingleton() {
    return singleton;
}

private StatsBuilder() {
    // NOTE: This one line is extremely important. The "custom" PMI
    // infrastructure is intended to create StatsInstances from XML
    // "template" (config) files, but in our case we want to create
    // them dynamically in code. To have the PMI infra look up something
    // dynamically if it can't find a template XML file we need to
    // register a "template lookup" class - this class.
    StatsFactory.registerStatsTemplateLookup(this);
}

public PmiModuleConfig getTemplate(String templateName) {
    if (logger.isDebugEnabled()) {
        logger.debug("getTemplate called for template of name ["
                     + templateName + "]");
    }
    return (PmiModuleConfig)templates.get(templateName);
}

public Object[] createTemplate(String beanName, Class beanClazz,
                               Integer[] statTypes, String setName) {
    String moduleId = beanName;
    if (templates.get(moduleId) != null) {
        if (logger.isInfoEnabled()) {
            logger.info("Statistics for bean of name [" + moduleId
                        + "] have already been generated. Skipping...");
        }
        return null;
    }

    PmiModuleConfig config = new PmiModuleConfig(moduleId);
    config.setDescription(beanClazz.getName());

    // In order to keep the noise down we monitor only public methods.
    Method[] methods = beanClazz.getMethods();

    List methodList = new java.util.ArrayList();

    int id = 0;

    for (int i = 0; i < j =" 0;" data =" getData(methods[i]," id =" id" interceptors =" new" instance =" StatsFactory.createStatsInstance(" i =" 0;" spistatistic =" instance.getStatistic(i);" methodid =" getMethodId(method);" typename =" StatsConstants.getTypeName(statType);" data =" new" setname ="=" methodname =" method.getName();" length ="=" methodid =" new" params =" method.getParameterTypes();" j =" 0;" paramclassname =" params[j].getName();" dotindex =" paramClassName.lastIndexOf('.');"> -1) {
            paramClassName = paramClassName.substring(dotIndex + 1);
        }

        methodID.append(paramClassName);

        if (j < params.length - 1) {
            methodID.append(',');
        }
    }
    methodID.append(')');

    return methodID.toString();
}
}

/**
* Simple listener class for statistic events.
* Note for future design: Maybe we want a StatisticActionListener to
* optionally log stats values to the commons-logging log file?
*/
class StatisticActionListenerImpl implements StatisticActionListener {
private Log logger = LogFactory.getLog(getClass());

public void statisticCreated(SPIStatistic spiStatistic) {
    if (logger.isDebugEnabled()) {
        logger.debug("Created statistic of type ["
                     + spiStatistic.getClass().getName() + "]");
    }
}

public void updateStatisticOnRequest(int i) {
    if (logger.isDebugEnabled()) {
        logger.debug("updateStatisticOnRequest called for statistic with id ["
                     + i + "]");
    }
}
}

StatsConstants.java


package robertmaldon.stats;

/**
* Some constants used in construction of PMI stats definitions.
*/
public final class StatsConstants {
public static final int TYPE_COUNT = 2;
public static final int TYPE_BOUNDED_RANGE = 5;
public static final int TYPE_RANGE = 7;
public static final int TYPE_TIME = 4;
public static final int TYPE_DOUBLE = 3;
public static final int TYPE_AVERAGE = 6;

public static final String TYPE_NAME_COUNT = "count";
public static final String TYPE_NAME_BOUNDED_RANGE = "boundedRange";
public static final String TYPE_NAME_RANGE = "range";
public static final String TYPE_NAME_TIME = "time";
public static final String TYPE_NAME_DOUBLE = "double";
public static final String TYPE_NAME_AVERAGE = "average";

public static final int LEVEL_LOW = 1;
public static final int LEVEL_MEDIUM = 3;
public static final int LEVEL_HIGH = 7;
public static final int LEVEL_MAX = 15;

private StatsConstants() {}

public static String getTypeName(int statType) {
switch (statType) {
  case TYPE_COUNT:
      return "@" + TYPE_NAME_COUNT;
  case TYPE_BOUNDED_RANGE:
      return "@" + TYPE_NAME_BOUNDED_RANGE;
  case TYPE_RANGE:
      return "@" + TYPE_NAME_RANGE;
  case TYPE_TIME:
      return "@" + TYPE_NAME_TIME;
  case TYPE_DOUBLE:
      return "@" + TYPE_NAME_DOUBLE;
  case TYPE_AVERAGE:
      return "@" + TYPE_NAME_AVERAGE;
  default:
      return "@unknown";
}
}

public static int getType(String typeName) {
if (TYPE_NAME_COUNT.equals(typeName)) {
  return TYPE_COUNT;
} else if (TYPE_NAME_BOUNDED_RANGE.equals(typeName)) {
  return TYPE_BOUNDED_RANGE;
} else if (TYPE_NAME_RANGE.equals(typeName)) {
  return TYPE_RANGE;
} else if (TYPE_NAME_TIME.equals(typeName)) {
  return TYPE_TIME;
} else if (TYPE_NAME_DOUBLE.equals(typeName)) {
  return TYPE_DOUBLE;
} else if (TYPE_NAME_AVERAGE.equals(typeName)) {
  return TYPE_AVERAGE;
} else {
  return 0;
}
}
}

AbstractStatsInterceptor.java


package robertmaldon.stats;

import com.ibm.wsspi.pmi.stat.SPIStatistic;

import org.aopalliance.intercept.MethodInterceptor;
import org.aopalliance.intercept.MethodInvocation;

/**
* Base class for our statistics around advice.
*/
public abstract class AbstractStatsInterceptor implements MethodInterceptor {

protected SPIStatistic spiStatistic;

public AbstractStatsInterceptor(SPIStatistic spiStatistic) {
this.spiStatistic = spiStatistic;
}

public abstract Object invoke(MethodInvocation invocation) throws Throwable;
}

TimeInterceptor.java


package robertmaldon.stats;

import org.aopalliance.intercept.MethodInvocation;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import com.ibm.wsspi.pmi.stat.SPITimeStatistic;

/**
* Average invocation time.
*/
public class TimeInterceptor extends AbstractStatsInterceptor {
public TimeInterceptor(SPITimeStatistic spiStatistic) {
super(spiStatistic);
}

public Object invoke(MethodInvocation invocation) throws Throwable {

SPITimeStatistic timeStat = (SPITimeStatistic)spiStatistic;

long start = System.currentTimeMillis();

Object rval = invocation.proceed();

long end  = System.currentTimeMillis();

long diff = end - start;

if (timeStat != null) {
  timeStat.add(diff);
}

return rval;
}
}

CountInterceptor.java


package robertmaldon.stats;

import org.aopalliance.intercept.MethodInvocation;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import com.ibm.wsspi.pmi.stat.SPICountStatistic;

/**
* Simple counter.
*/
public class CountInterceptor extends AbstractStatsInterceptor {
public CountInterceptor(SPICountStatistic spiStatistic) {
super(spiStatistic);
}

public Object invoke(MethodInvocation invocation) throws Throwable {

SPICountStatistic countStat = (SPICountStatistic)spiStatistic;

Object rval = invocation.proceed();

if (countStat != null) {
  countStat.increment();
}

return rval;
}
}

Update: I've had a couple of requests for a ZIP of the source code, so here it is.

robertmaldon

Thursday, November 30, 2006

Temporary Email Address Reduces Spam

Wednesday, November 15, 2006

Sun Java implementation goes open source under the GPL!

Friday, November 10, 2006

Windows Blue Screen of Death (BSOD) - the screensaver!

Thursday, November 09, 2006

Integrating WebSphere PMI and Spring