The IT Measurement Inversion

BEFORE YOU MAKE A MULTIMILLION-DOLLAR investment in IT, you probably do a cost-benefit analysis. Of all the variables on that spreadsheet, which do you spend the most time and effort trying to estimate? Most organizations spend a lot of time estimating initial development costs but not so much estimating benefits, how fast people will start using the system or the chance the project will never be finished. If you calculate the economic value of additional information you need for more accurate ROI estimates, you see that the typical measurement priorities are often inappropriate. In fact, the variables that are most valuable for reducing uncertainty in ROI estimation are rarely represented on IT cost-benefit spreadsheets.

I call this paradox the “IT measurement inversion.” I detected this effect as my client was rolling out a new IT decision-making methodology based on applied information economics (AIE). One distinctive feature of this approach is that it includes a measure of the degree of uncertainty for each variable in an ROI analysis. This gives us the foundation for another AIE figure: the economic value of information.

Measuring Uncertainty

Almost every variable in a cost-benefit analysis is uncertain. We don’t know exactly what initial costs will be or how much an improvement in productivity will yield. Yet typically in the analysis of IT investments, every cost or benefit is shown as a single, precise number. This implies that the exact number is known, which is almost never the case. It is possible, however, to represent those values more realistically. We can use “probability distributions” to show how much we really know about each number in a cost-benefit analysis. Quantifying uncertainty is a prerequisite to calculating the value of information.

You may vaguely recall concepts like “90 percent confidence interval” or “standard deviation” from a college statistics class. Though your math skills may have faded, the concepts are fairly simple. A 90 percent confidence interval consists of two numbers-an upper and a lower bound-that represent a range that we can be 90 percent sure contains the true value. For training costs of a new system, most IT cost-benefit spreadsheets would simply use one number, such as $500,000. But perhaps all we can say is that we are 90 percent sure the training costs will be between $300,000 and $700,000. There is still a 10 percent chance that the real number will fall outside of bounds. We can use statistical representations (such as normal distributions, discrete binary and so forth) to determine the amount of uncertainty. Applying a normal distribution (not always the most realistic but the most straightforward example) to our scenario, we could use the standard deviation to express the uncertainty. According to statistical mathematics, there are 3.29 standard deviations in a 90 percent confidence interval. Another way to say this is training costs are a normally distributed number with a mean of $500,000 and a standard deviation of $121,580-the range ($400,000) divided by 3.29. Every uncertain number in the ROI analysis should be represented as a distribution to show its uncertainty. Once we have quantified the uncertainty of each variable, we can start calculating how much additional information is worth for each variable.

The Value of Information

Information has value because it leads us to better decisions. The method for calculating the value of information has been around for decades, an offshoot of game theory and decision theory. Roughly put, the value of information equals the chance of being wrong times the cost of being wrong. The cost of being wrong-that is, what is lost if your decision doesn’t work out-is called an opportunity loss. For a simplistic example, say you’re considering investing $1 million in a new system. It promises a net $3 million gain over three years. (For our example’s sake, it’ll either be completely successful or a total bomb.) If you invest but the system fails, your mistake costs you $1 million. If you decide not to invest and you should have, the mistake costs you $3 million. When we multiply the opportunity loss by the chance of a loss, we get the expected opportunity loss (EOL). Calculating the value of information boils down to determining how much it will reduce EOL.

The economic value of a variable in the cost-benefit analysis essentially tells how much the additional information about that variable reduces the EOL for the investment. Information about a variable that reduces the EOL by a lot strongly affects whether you pursue the investment. This is a sensitivity analysis of the decision that gives a dollar value for the information.

Even in projects with very uncertain development costs, we haven’t found that those costs have a significant information value for the investment decision. In other words, information about the development costs did not lower the EOL as much as did information about other variables. The most important variables-those with the most uncertainty and impact on the decision-are rarely represented in cost-benefit analysis.

Truly Important Variables

The single most important unknown is whether the project will be canceled. This is a binary phenomenon-it either happens or it doesn’t-and so is represented with a single percentage probability (not a range). If cancellation occurs, no benefits are ever received; but at least some development costs accrue.

The next most important variable is utilization of the system, including how quickly the system rolls out and whether some people will use it at all. Like cancellation, uncertainty about how soon people use the system never shows up on most ROI calculations. Many simply assume that benefits begin upon implementation, with no allowance for ramp-up or difficulties.

Of course you need to estimate development costs when making a decision about IT investments. However, you don’t usually need to reduce the uncertainty about those costs to make an informed decision. Reducing the uncertainty about the utilization rate and the likelihood of cancellation of a new system is much more important when deciding how to spend your money.

Beat the Paradox

Why the IT measurement inversion? Perhaps we focus on measuring what we feel comfortable measuring regardless of how important it is. The IT industry also emphasizes development costs; there are a lot of “software metrics” firms that focus on measuring development costs, to the exclusion of other more important variables. Another reason for the paradox may be simpler. Many people simply don’t know there is a formula for the economic value of information.

Counteract measurement inversion by measuring the value of information. First, learn how to model uncertainties in ROI calculations. Start doing this as soon as possible, especially on your largest IT investments. Your organization may already have people with the statistical training to do this.

Second, for the investment decision, don’t bother hiring software metrics services to reduce initial uncertainty about costs. A range of, say, $2 million to $4 million for an IT investment may seem very uncertain, but it is probably less so than some of the benefits. You may need to count function points later for managing programmers’ productivity, but spend your money and efforts to reduce other uncertainties first. And be sure to consider costs for systems maintenance.

Third, take what you would have spent on software metrics and focus it on researching the system’s benefits, how quickly people will start using the system, how much it will be used and what the risk of cancellation is.

To make the best investment decisions, IT management must learn how to compute the value of information, and start modeling utilization and cancellation in cost-benefit arguments. Once they do, measurements will be properly prioritized and the IT measurement inversion will disappear.

Douglas Hubbard recently founded Hubbard Ross LLC in Glen Ellyn, Ill. He can be reached via e-mail at mailto:dwhubbard@hubbardresearch.com.