CIO — BEFORE YOU MAKE A MULTIMILLION-DOLLAR investment in IT, you probably do a cost-benefit analysis. Of all the variables on that spreadsheet, which do you spend the most time and effort trying to estimate? Most organizations spend a lot of time estimating initial development costs but not so much estimating benefits, how fast people will start using the system or the chance the project will never be finished. If you calculate the economic value of additional information you need for more accurate ROI estimates, you see that the typical measurement priorities are often inappropriate. In fact, the variables that are most valuable for reducing uncertainty in ROI estimation are rarely represented on IT cost-benefit spreadsheets.

I call this paradox the "IT measurement inversion." I detected this effect as my client was rolling out a new IT decision-making methodology based on applied information economics (AIE). One distinctive feature of this approach is that it includes a measure of the degree of uncertainty for each variable in an ROI analysis. This gives us the foundation for another AIE figure: the economic value of information.

Measuring Uncertainty

Almost every variable in a cost-benefit analysis is uncertain. We don't know exactly what initial costs will be or how much an improvement in productivity will yield. Yet typically in the analysis of IT investments, every cost or benefit is shown as a single, precise number. This implies that the exact number is known, which is almost never the case. It is possible, however, to represent those values more realistically. We can use "probability distributions" to show how much we really know about each number in a cost-benefit analysis. Quantifying uncertainty is a prerequisite to calculating the value of information.

You may vaguely recall concepts like "90 percent confidence interval" or "standard deviation" from a college statistics class. Though your math skills may have faded, the concepts are fairly simple. A 90 percent confidence interval consists of two numbers-an upper and a lower bound-that represent a range that we can be 90 percent sure contains the true value. For training costs of a new system, most IT cost-benefit spreadsheets would simply use one number, such as $500,000. But perhaps all we can say is that we are 90 percent sure the training costs will be between $300,000 and $700,000. There is still a 10 percent chance that the real number will fall outside of bounds. We can use statistical representations (such as normal distributions, discrete binary and so forth) to determine the amount of uncertainty. Applying a normal distribution (not always the most realistic but the most straightforward example) to our scenario, we could use the standard deviation to express the uncertainty. According to statistical mathematics, there are 3.29 standard deviations in a 90 percent confidence interval. Another way to say this is training costs are a normally distributed number with a mean of $500,000 and a standard deviation of $121,580-the range ($400,000) divided by 3.29. Every uncertain number in the ROI analysis should be represented as a distribution to show its uncertainty. Once we have quantified the uncertainty of each variable, we can start calculating how much additional information is worth for each variable.

The Value of Information

Information has value because it leads us to better decisions. The method for calculating the value of information has been around for decades, an offshoot of game theory and decision theory. Roughly put, the value of information equals the chance of being wrong times the cost of being wrong. The cost of being wrong-that is, what is lost if your decision doesn't work out-is called an opportunity loss. For a simplistic example, say you're considering investing $1 million in a new system. It promises a net $3 million gain over three years. (For our example's sake, it'll either be completely successful or a total bomb.) If you invest but the system fails, your mistake costs you $1 million. If you decide not to invest and you should have, the mistake costs you $3 million. When we multiply the opportunity loss by the chance of a loss, we get the expected opportunity loss (EOL). Calculating the value of information boils down to determining how much it will reduce EOL.

The economic value of a variable in the cost-benefit analysis essentially tells how much the additional information about that variable reduces the EOL for the investment. Information about a variable that reduces the EOL by a lot strongly affects whether you pursue the investment. This is a sensitivity analysis of the decision that gives a dollar value for the information.

Even in projects with very uncertain development costs, we haven't found that those costs have a significant information value for the investment decision. In other words, information about the development costs did not lower the EOL as much as did information about other variables. The most important variables-those with the most uncertainty and impact on the decision-are rarely represented in cost-benefit analysis.

Continue Reading