Sources Of IT Metric Data

Where can I.T. managers get I.T. metrics data from today?
There are over 40 possible sources, which can be roughly grouped into four categories:

• Consulting Firms
• Industry Groups
• Publications
• Software Firms

Some sources are expensive, some are cheap. Some are authoritative, some can be in accurate. There are no sources which are Authoritative, Affordable, Accurate, and Comprehensive.

Consulting Firms

Tier 1 Benchmarkers

These are individually commissioned and priced benchmarking projects in which the benchmarker compares the customer's metrics with data from a database built from previous benchmarks. Such projects typically take 8 to 12 weeks. Forrester Research say they can cost up to $200k. There are four main players:

• Gartner Market leader, 20 year track record. Global reach, though recently closed their Australian benchmarking practice.
• Compass 20 year track record. Global reach.
• Maturity. Newer player, founded in 2002 in Germany.
• Nautilus Another newer player, founded in the US in 2005. Offer innovative Pro-Benchmark tool.

Other Benchmarkers & Consultancies

There are a number of smaller players who carry out benchmarking work within specific IT departments, or in some countries only.

• Hackett Group Atlanta GA based Hackett have offered some form of benchmarking for many years, though they have not competed vigorously with the larger commercial benchmarkers. Used to offer a "Book Of Numbers", but now seem to offer a no-fee 45 to 60 minute benchmark, resulting in a report comparing your performance against others in their database.
• Global Information Partners. Small US-based benchmarker associated with IMF (see below)
• Metri are a European benchmarking company started in 2005
• Hedra Consulting provide a benchmarking service largely aimed at UK government organisations through IT Perceptions
• Lexta Consulting in Germany. This page is in German but can be translated via Google
• IDC Australia have announced a benchmarking service in Australia in 2008
• Metric's Based Assessments Mainframe and server cost benchmarks, mainly US NE seaboard, experienced ex META benchmarker.
• David Consulting Application Development/Function Points only
• QPMG Application Development/Function Points only
• SPR Application Development/Function Points only, associated with Capers Jones, a leading thinker on software engineering productivity.

Industry Groups

Industry Process Models

• ITIL, the Information Technology Infrastructure Library, is a set of books and training courses describing the major processes and best practices in running an IT organisation. It was written by volunteers from the IT Service Management Forum under the sponsorship of the UK government, but has been adopted increasingly internationally, leading to an update and re-write to version 3 in 2008. ITIL believes strongly in measurement and metrics, and suggests a few for most processes without ever going into detail about how to collect and use them. In 2006, ITSMF published a supplementary book, [http://www.amazon.com/Metrics-Service-Management-Peter-Brooks/dp/9077212698 Metrics for I.T. Service Management by Peter Brooks, which describes 228 metrics structured around the old ITIL version 2 processes. An initiative in The Netherlands is using these metrics to enable customers to benchmark their ITIL performance. There are a number of other ITIL compliance assessment services offered, of which the most well established is ISO 20000, a formal ISO standard for compliance with best practice in managing I.T. Also investigate this book. The strength of the ITIL approach is in describing processes, and most ITIL-based guides to metrics measure processes, rather than day to day activities. Thus, for example, the Peter Brooks book defines 15 metrics for a process for managing Availability, but it doesn't describe the practical differences in implementing 99% availability compared with 99.999%, or how to calculate the extra cost of hardware, software, and people needed to achieve this.
• COBIT. Widely adopted best practice and process guide which in v4 includes process metrics definition for over 300 processes. Plans for benchmarking, but not sure of the status.
• CMM, CMM-I Process maturity scorecard aimed mainly at software development and system integration.
• IFPUG Application Development only - industry body specifying Function Point counting rules
• Benchmark Portal is a service run through Purdue University which benchmarks contact centres, offering a free "Reality Check", reports on industry performance levels, and a specific peer comparison service. Note that this covers on contact centres, and is not restricted simply to IT Help Desks.

Other Industry Bodies

• AQPC. US based not-for-profit organisation offering high to medium level list of recommended IT metrics.
• IMF US based CIO trade federation (of which HP is a member!) which includes a benchmarking service for members. Experienced ex-Compass benchmarking experts.
• SOCITM Society of UK local government IT managers who regularly benchmark on 10-15 metrics
• SPEC. Industry body providing standard performance ratings of CPU/motherboard configurations
• TPC. Industry body providing standard performance ratings of servers running (mostly) database workloads.
• Uptime Institute

Publications

Publications are books or libraries which can be bought for a one-off or a monthly subscription fee.

Industry Analyst Firms

• Gartner TCO Studies. In addition to their benchmarking research, Gartner introduced the term Total Cost Of Ownership (TCO) in 1993 and still predict TCO figures in their research papers. They used to offer a software tool to calculate it, but this was retired in 2006 and is no longer available.
• Gartner Key Metrics Reports. Additionally, Gartner offer an annual publication which used to be called the Worldwide Benchmark Report (published by META Group), which reports typical ranges for some IT costs and staffing levels. The report is believed to cost in the region of US$5,000 per year.
• Forrester Research. There are one or two research notes available on metrics from Forrester, but many IT functions are not covered, and many of the notes are extremely [http://www.forrester.com/Research/PDF/0,5110,28758,00.pdf old (2002 or 2003). Only available to paid-up Forrester subscribers.

Other Publications

• Open Source TCO Model. Little used Swiss-run open source TCO software.
• Computer Economics. Web site selling publications.
• Jochen Mikels runs a German language web site selling market prices
• Peter Brooks's book on Metrics For Service Management. Book based on ITIL v2 processes, defining 100+ metrics for ITIL process management, many impractical. No target data published, but see also ITSMF Benchmarking.
• Implementing Metrics For Service Management by David Smith
• ISBGS/UKSMA CD-based report on software development project productivity using Function Point counts.
• ITMPI Sponsored by Application Development outsourcer to document best practices.
• ITMWeb. IT Managers web resource, not updated since 2003.
• Janco's IT Productivity Center offer standard job descriptions and US based salary data for various IT roles. One of their products is a Metric's Handiguide (304 pages, last updated August 2007) containing lists of 504 metrics, including graphs, for US$599. It's not clear from the sample how much detail is provided on the definition of the metrics, or any indication of good/bad performance. However an "update" service for $379 offers "12 months of updates".
• See also David Parmenter's web site and his resources at Better Management
• Computer Economics book

Software Vendors

Software vendors tend to talk about the metrics their product can deliver, but don't go so far as to provide either definitions or data for these.

The most interesting effort is from Mirror 42 who have also started a free library of 367 IT metrics. They give brief definitions only, little detail. Little or no data. Strong orientation on process metrics.

Other references to IT metrics from software vendors can be found at:

• BMC
• Pureshare
• Cognos
• Digital Fuel