Monday, September 18, 2006

Challenges of running a Managed Service

Over recent years I've seen quite a few companies, particularly IBs, create their own IT Managed Services. They are interesting beasts, and I learnt quite a but about them while working inside a large IB that established a Managed Service based on a J2EE infrastructure. Below are some of my observations of these beasts.

What is a Managed Service?

A Managed Service assumes responsilibity of day-to-day running of important business critical systems. The same Managed Service group can assume responsibility for a number of applications that cross multiple business lines.

The main purpose of a Managed Service is to save money by centralizing operations and infrastructure instead of having to duplicate the same stuff for every new application.

A Managed Service model is very similiar to an ASP or help-desk model, but differs in that it is usually internally run (i.e. not outsourced, but many members of the group may still be Inidian consultants) and closely aligned with the business.

What are the incentives for starting/running/using a managed service?

The main incentives to run a managed service are:
  • Reduced cost by hosting as many applications as possible on common servers (i.e. less servers required to host the same number of applications than if each business group had their own servers)
  • Reduced number of staff required to support the infrastructure (i.e. each business group does not need to have its own support teams)
  • Reduced cost by using common infrastructure (i.e. developers do not have to keep "re-inventing the wheel" every time)
  • You do not have the resources to deploy and support applications, so you are happy to use somebody else's infrastructure (as long as the price is right!)
What are some of the barriers to starting a managed service?
  • Control. IT people in particular like to have control over their infrastructure and will often re-invent the wheel.
  • My empire is bigger than yours. A managed service is likely to be used by many different business groups and therefore has the potential to be larger than a group within one business line. Managers within the business lines may get very jealous.
Keeping it Visible

The most important thing you can do to establish and maintain a managed service is to keep the service visible. i.e. tell everybody about the service and keep telling them.

If people don't know about the managed service they won't use it. If people become complacent about the service they will stop using it.

Some ways you can keep a managed service visible are:

  • Regular presentations to managers and developers.
  • Comparison of the managed service with similiar services. It is often hard to get information on managed services being run inside competing companies; it is easier to get information on services provided by application hosting companies. The most important comparisons are cost and Service Level Agreements (SLAs).
  • Statistics. Managers need to know how the managed service is being used, including:
    • Number of applications deployed
    • Number of outages
    • Resources used by the applications (memory, CPU, disk space, etc)
  • Provide feedback mechanisms. Customers need to feel they are being listened to.

Depending on how much money you have it is generally standard practice to have at least the following three environments:

Environment Description
Development A development environment should be similiar to a production environment, but developers should be given the freedom to experiment with it. With freedom comes the potential for disaster. If possible isolate one development environment from another. Make it clear to developers that support for development environments is much less than support for QA and production environments. The worst case should be you may have to destroy and then recreate the environment from scratch.
Quality Assurance (QA) The QA environment should be as similiar to production as possible. This means you follow the same processes to get an application into the QA environment as you follow to get the application into the production environment. Access to the QA environment should be restricted to the managed service support staff.
Production The processes for deploying an application into production must be well defined, well understood and - above all else - auditable. Auditability simply means all changes to an environment (e.g. a change to configuration or a new version of an application is installed) have been recorded and are reproduceable. This often means changes are scripted and those scripts are put under version control. Access to the production environment must be restricted to the managed service support staff.

Management, Management, Management

Some of the key things you need to manage in a managed service:
  • Well-defined Change Management Processes. The processes of getting an application installed into a managed environment must be well documented and well communicated.
  • Change Windows. It must be well reported when changes can be applied to a managed environment. Changes to a production environment have to typically be done out-of-business-hours, such as late at night and at weekends. The processes to get your application into a change window must be well documented.
  • Escalation Procedures. Formal procedures to raise issues is a must. If you don't have any escalation procedures then people will do all sorts of ad-hoc and chaotic things to work around problems; often an issue will be reported by a person up through their management chain, then down through your management chain.
  • Request / Bug Tracking system. A system to track work requests and bugs gives the impression of a professionally run managed service.
  • Patching Policy. One of the trickiest issues to deal with is the issue of patching. Let us say you are running 10 applications on the same server using the same libraries. One of the applications report a bug. A bug fix is made to one of the libraries. The dilema: Do you apply the patch required by the one application to the libraries used by all 10 applications? Do you do regular (e.g. six montly) patch updates to your managed service? If you don't do at least regular updates then your infrastructure becomes difficult to support; big hardware and software vendors do regular updates to their products and can be slow to fix old versions.
  • The Stick. People must be given incentive to not abuse the managed service infrastructure. There must also be incentive to agree with the patching policy. The only incentive that really works is to charge people more money if they don't follow the managed service guidelines.
Don't forget the developers!!!

Starting a managed service and continued funding for a managed service will require the support of senior managers within the company, so they should obviously the primary target of your efforts. Developers will (most of the time) use whatever their managers tell them to use.

However, don't neglect the developers. If the developers are not happy with a managed service then they can spread rumours. And some developers will become managers in the future.

Developers are usually interested in the technical details and the processes. Ways to keep developers happy include:

  • Simple, brief "Getting Started" documentation. This includes doco on:
    • Guidelines on use of the supported technologies (e.g. "this is how you use Message Driven Beans inside WebSphere over MQ")
    • Using various development tools
    • Processes to build and deploy applications in the managed environment
    • Tricks and gotchas
    An internal website is a good place for this stuff.
  • Choice. Developers like to experiment. While a managed service must lock down processes for QA and production environments, you must allow a bit more freedom in the development process. For example, you can recommend and officially support a particular development tool, but you shouldn't try to force developers to use those tools.
  • Technology chat channels. Get into the 21st century and install an instant messaging system. Developers love to share insights and opinions with other developers.
  • Access to debug information. Since developers won't have open access to QA and production environments they will need access to some sort of debug information so they can support the applications. At a minimum they will need secure access to application log files.
  • Request / Bug Tracking system. The ability for developers/managers to sumbit requests for work and bug reports. The submitter can access the system to check the progress of their request, and review the work that was done to complete previous requests.

No comments: