When reading up on a new technology, the last thing CIOs have time for is drilling down into the methodologies analysts use in preparing their forecasts and surveys. The CIO glances at the paragraph summarizing how many people participated in the survey, or she scans some of the basic forecasting assumptions, and that’s about it. And to some degree, that’s exactly what the analyst firms are counting on. The fact is, most analysts’ methodologies are something less than scientific. Just how much less than scientific is a matter of some mystery. Analysts don’t like to reveal what’s going on behind the curtain. If you are curious, however, don’t be afraid to push them on this.“If you’re paying for the research, you have every right to know how those numbers are generated. They usually say, ’Well, we have this model that we use.’ You can demand to walk through every single algorithm if you want to,” says Scott McCready, a longtime analyst and now president of CIOview, a software vendor in Boxborough, Mass.Mike Conlon, CIO at the University of Florida Health Science Center in Gainesville, just happens to also be a biostatistician. In fact, Conlon has a PhD in the application of statistical principles to medical research. And when he analyzes analyst methodologies, he doesn’t like what he sees. For example, Conlon takes issue with the way his analyst firm (Gartner) uses certain terms that have statistical relevance, such as probability (as in this invented example: “The majority of IT projects are destined for failure: 0.9 probability”). “These probability statements are not probability statements as any statistician would be used to seeing them,” says Conlon. “Their use of the term has nothing to do with mathematical models and scientific methodology. It’s jargon. What they mean to show is how confident they are in their prediction.”Jamie Popkin, Gartner group vice president and research fellow, explains that the company’s probability statements indicate Gartner’s level of confidence that an event will occur within a certain time frame. So why does Gartner dress them up in a veneer of statistical authority? The probability statements aren’t meant to convey scientific certitude?they’re just an easy shorthand for describing confidence levels, Popkin says.Conlon argues that the statements may be misleading to those not trained in statistics. But to be fair, he says, it would probably be impossible to bring statistical rigor to the world of technology forecasting because the forecast would have to be based on too many assumptions to be valid. Remember that the next time you’re holding a forecast in your hand, no matter how “confident” its author. Related content brandpost Sponsored by SAP When natural disasters strike Japan, Ōita University’s EDiSON is ready to act With the technology and assistance of SAP and Zynas Corporation, Ōita University built an emergency-response collaboration tool named EDiSON that helps the Japanese island of Kyushu detect and mitigate natural disasters. By Michael Kure, SAP Contributor Dec 07, 2023 5 mins Digital Transformation brandpost Sponsored by BMC BMC on BMC: How the company enables IT observability with BMC Helix and AIOps The goals: transform an ocean of data and ultimately provide a stellar user experience and maximum value. By Jeff Miller Dec 07, 2023 3 mins IT Leadership brandpost Sponsored by BMC The data deluge: The need for IT Operations observability and strategies for achieving it BMC Helix brings thousands of data points together to create a holistic view of the health of a service. By Jeff Miller Dec 07, 2023 4 mins IT Leadership how-to How to create an effective business continuity plan A business continuity plan outlines procedures and instructions an organization must follow in the face of disaster, whether fire, flood, or cyberattack. Here’s how to create a plan that gives your business the best chance of surviving such an By Mary K. Pratt, Ed Tittel, Kim Lindros Dec 07, 2023 11 mins Small and Medium Business IT Skills Backup and Recovery Podcasts Videos Resources Events SUBSCRIBE TO OUR NEWSLETTER From our editors straight to your inbox Get started by entering your email address below. Please enter a valid email address Subscribe