News Feature

Avoiding Unreliable Metrics

By Jack Keen

October 01, 2003 — CIO — I confess. I’m a numbers kind of guy. But I’m concerned. I see too many numbers, with too little meaning, cavalierly factored into too many IT value discussions. Metrics are often management’s single most important determinant of value. So allowing anything less than top quality numbers—and the correct context for these measurements—leads to wrong decisions about which IT investments to select, which implementation paths to choose or what portfolios to manage.

There are three common ways that people misuse metrics: tolerating ambiguity, discouraging guesstimates and overlooking available sources. Let’s take a look at some solutions for avoiding these missteps.

Tolerating Ambiguity Numbers falsely convey a mystique of truth, by their very existence. But rarely do numbers actually mean what they seem to imply. That is because a metric is not "the result." It is shorthand for a story about the result. If the metric user leaves out the story portion of the metric, we are left to make potentially erroneous assumptions to fill in the gaps. For example, if someone states "we have a 25 percent customer retention rate," he is providing us with an ambiguous metric. Because the story behind the metric is missing, his message has no infallible meaning. In this example, a good metric story might say: "Our internal audit committee confirmed last April that our Southern division has experienced a 25 percent customer retention rate, for those customers who have bought at least $10,000 annually from us, for the past three years in a row. Sue Smith, vice president of finance, has stated that this retention rate is too low to allow us to realize the gross margin goals for this year." Armed with that explanation, we are less likely to misuse the metric’s meaning and implications, and can make an informed decision.

Discouraging Guesstimates Another misuse of numbers occurs when management refuses to accept guesstimates (informed estimates) as legitimate metrics. This problem occurs whenever someone says, "If you can’t prove it, we won’t use it." In this mistaken worldview, only hard metrics from factual situations are valid. The clear message is that guesses don’t count. The reality is that informed guesses are what makes the world work. If business investment strategies were 100 percent fact-based, we wouldn’t need high-price executives to guide the company. Computers could probably do the job via fancy calculations. The whole discipline of risk management is ultimately grounded in probabilities, which themselves are estimates. The medical profession begins and ends with the reliability of doctor diagnoses, none of which are 100 percent correct. But they get enough right that we don’t ban their profession. A similar situation exists with our economy’s reliance on meteorologists. They don’t hit the mark 100 percent of the time. But their track record is good enough for airplanes to fly safely through troubling storms, not to mention properly attiring our families for school and work. The key to accepting informed guesses as valid metrics is to make sure they: A. Come from knowledgeable, experienced and dependable people; and B. Are accompanied by clear explanations of the guesstimate’s premises, assumptions and logic.