Analyzing IT Analyst Services Firms: What’s Behind Their Numbers

When reading up on a new technology, the last thing CIOs have time for is drilling down into the methodologies analysts use in preparing their forecasts and surveys. The CIO glances at the paragraph summarizing how many people participated in the survey, or she scans some of the basic forecasting assumptions, and that’s about it. And to some degree, that’s exactly what the analyst firms are counting on.

The fact is, most analysts’ methodologies are something less than scientific. Just how much less than scientific is a matter of some mystery. Analysts don’t like to reveal what’s going on behind the curtain. If you are curious, however, don’t be afraid to push them on this.

“If you’re paying for the research, you have every right to know how those numbers are generated. They usually say, ’Well, we have this model that we use.’ You can demand to walk through every single algorithm if you want to,” says Scott McCready, a longtime analyst and now president of CIOview, a software vendor in Boxborough, Mass.

Mike Conlon, CIO at the University of Florida Health Science Center in Gainesville, just happens to also be a biostatistician. In fact, Conlon has a PhD in the application of statistical principles to medical research. And when he analyzes analyst methodologies, he doesn’t like what he sees.

For example, Conlon takes issue with the way his analyst firm (Gartner) uses certain terms that have statistical relevance, such as probability (as in this invented example: “The majority of IT projects are destined for failure: 0.9 probability”). “These probability statements are not probability statements as any statistician would be used to seeing them,” says Conlon. “Their use of the term has nothing to do with mathematical models and scientific methodology. It’s jargon. What they mean to show is how confident they are in their prediction.”

Jamie Popkin, Gartner group vice president and research fellow, explains that the company’s probability statements indicate Gartner’s level of confidence that an event will occur within a certain time frame. So why does Gartner dress them up in a veneer of statistical authority?

The probability statements aren’t meant to convey scientific certitude?they’re just an easy shorthand for describing confidence levels, Popkin says.

Conlon argues that the statements may be misleading to those not trained in statistics. But to be fair, he says, it would probably be impossible to bring statistical rigor to the world of technology forecasting because the forecast would have to be based on too many assumptions to be valid.

Remember that the next time you’re holding a forecast in your hand, no matter how “confident” its author.