Avoiding Unreliable Metrics

I confess. I’m a numbers kind of guy. But I’m concerned. I see too many numbers, with too little meaning, cavalierly factored into too many IT value discussions. Metrics are often management’s single most important determinant of value. So allowing anything less than top quality numbers—and the correct context for these measurements—leads to wrong decisions about which IT investments to select, which implementation paths to choose or what portfolios to manage.

There are three common ways that people misuse metrics: tolerating ambiguity, discouraging guesstimates and overlooking available sources. Let’s take a look at some solutions for avoiding these missteps.

Tolerating Ambiguity Numbers falsely convey a mystique of truth, by their very existence. But rarely do numbers actually mean what they seem to imply. That is because a metric is not “the result.” It is shorthand for a story about the result. If the metric user leaves out the story portion of the metric, we are left to make potentially erroneous assumptions to fill in the gaps. For example, if someone states “we have a 25 percent customer retention rate,” he is providing us with an ambiguous metric. Because the story behind the metric is missing, his message has no infallible meaning. In this example, a good metric story might say: “Our internal audit committee confirmed last April that our Southern division has experienced a 25 percent customer retention rate, for those customers who have bought at least $10,000 annually from us, for the past three years in a row. Sue Smith, vice president of finance, has stated that this retention rate is too low to allow us to realize the gross margin goals for this year.” Armed with that explanation, we are less likely to misuse the metric’s meaning and implications, and can make an informed decision.

Discouraging Guesstimates Another misuse of numbers occurs when management refuses to accept guesstimates (informed estimates) as legitimate metrics. This problem occurs whenever someone says, “If you can’t prove it, we won’t use it.” In this mistaken worldview, only hard metrics from factual situations are valid. The clear message is that guesses don’t count. The reality is that informed guesses are what makes the world work. If business investment strategies were 100 percent fact-based, we wouldn’t need high-price executives to guide the company. Computers could probably do the job via fancy calculations. The whole discipline of risk management is ultimately grounded in probabilities, which themselves are estimates. The medical profession begins and ends with the reliability of doctor diagnoses, none of which are 100 percent correct. But they get enough right that we don’t ban their profession. A similar situation exists with our economy’s reliance on meteorologists. They don’t hit the mark 100 percent of the time. But their track record is good enough for airplanes to fly safely through troubling storms, not to mention properly attiring our families for school and work. The key to accepting informed guesses as valid metrics is to make sure they: A. Come from knowledgeable, experienced and dependable people; and B. Are accompanied by clear explanations of the guesstimate’s premises, assumptions and logic.

Overlooking Available Sources For many CIOs, the most distressing situation occurs with metric misuse when the really good, hard, relevant ones exist, but they go undiscovered. More often than not, this occurs from lack of elementary research skills by those charged with finding such metrics. In this age of information overload, useful metrics, of direct validity for the company’s situation, may well exist, yet are hidden within readily accessible sources—such as the publications from trade associations, research by business schools and surveys from industry analysis firms, to name a few. Ironically, sometimes an organization’s customers either have the missing metrics or would be willing to set up a program to find them, but no one has asked them to do it.

These are all metrics missteps that routinely derail projects or lead to costly and incorrect decisions. But using numbers correctly is a skill that can be learned. Read on for some specific advice on how to help your enterprise use numbers in more powerful ways.

Five Steps to Ensure Reliable Metrics

You can use several methods to make your staff and management aware of the pitfalls of misusing numbers. Here are some ways to strengthen your organization’s metric savvy-ness:

Conduct Sensitivity Education Alert both management and staff to the promises and pitfalls of metric usage. Help them to become more sensitive to metric fumbles, as well as home runs. Assist hard-core numbers people in getting more comfortable with solid guesstimates built on informed analysis.

Apply the Metric Trustworthiness Test Apply this three-way test to all metrics: 1. Is the metric clearly defined and explained? 2. Is the source of the metric visible and credible? 3. Is the metric’s story readily available and relevant to our situation? Trustworthy metrics score three out of three on this test.

Actively Seek Industry and Vendor Help Identify industry trade associations, industry analysts and vendors that consistently pass the Metrics Trustworthiness test. Tell them what types of metrics you are looking for, then ask for their help in finding them.

Do Internet Research Aggressively Find relevant metrics from studies by others. Try searching the Internet, using keywords such as metrics and a topic. For example, a Google search using the keywords metrics and help desk brings a wealth of information for a help desk business case.

Publish Internal Guidelines Publicize the organization’s view concerning what types of metrics are valid, why and where to find them.

Projecting a healthy skepticism toward numbers dealing with IT value can help turn metric disrespect into metric success. Like many good things in life, however, success at “metricification” requires a modicum of education, discipline and persistent follow-up. Fortunately, it’s worth it.