When reading up on a new technology, the last thing CIOs have time for is drilling down into the methodologies analysts use in preparing their forecasts and surveys. The CIO glances at the paragraph summarizing how many people participated in the survey, or she scans some of the basic forecasting assumptions, and that’s about it. And to some degree, that’s exactly what the analyst firms are counting on. The fact is, most analysts’ methodologies are something less than scientific. Just how much less than scientific is a matter of some mystery. Analysts don’t like to reveal what’s going on behind the curtain. If you are curious, however, don’t be afraid to push them on this.“If you’re paying for the research, you have every right to know how those numbers are generated. They usually say, ’Well, we have this model that we use.’ You can demand to walk through every single algorithm if you want to,” says Scott McCready, a longtime analyst and now president of CIOview, a software vendor in Boxborough, Mass.Mike Conlon, CIO at the University of Florida Health Science Center in Gainesville, just happens to also be a biostatistician. In fact, Conlon has a PhD in the application of statistical principles to medical research. And when he analyzes analyst methodologies, he doesn’t like what he sees. For example, Conlon takes issue with the way his analyst firm (Gartner) uses certain terms that have statistical relevance, such as probability (as in this invented example: “The majority of IT projects are destined for failure: 0.9 probability”). “These probability statements are not probability statements as any statistician would be used to seeing them,” says Conlon. “Their use of the term has nothing to do with mathematical models and scientific methodology. It’s jargon. What they mean to show is how confident they are in their prediction.”Jamie Popkin, Gartner group vice president and research fellow, explains that the company’s probability statements indicate Gartner’s level of confidence that an event will occur within a certain time frame. So why does Gartner dress them up in a veneer of statistical authority? The probability statements aren’t meant to convey scientific certitude?they’re just an easy shorthand for describing confidence levels, Popkin says.Conlon argues that the statements may be misleading to those not trained in statistics. But to be fair, he says, it would probably be impossible to bring statistical rigor to the world of technology forecasting because the forecast would have to be based on too many assumptions to be valid. Remember that the next time you’re holding a forecast in your hand, no matter how “confident” its author. Related content BrandPost The future of trust—no more playing catch up Broadcom: 2023 Tech Trends That Transform IT By Eric Chien, Director of Security Response, Symantec Enterprise Division, Broadcom Mar 31, 2023 5 mins Security BrandPost TCS gives Blackhawk Network an edge with Microsoft Cloud In this case study, Blackhawk Network’s Cara Renfroe joins Tata Consultancy Services’ Rakesh Kumar and Microsoft’s Nilendu Pattanaik to explain how TCS transformed the gift card company’s customer engagement and global operati By Tata Consultancy Services Mar 31, 2023 1 min Financial Services Industry Cloud Computing IT Leadership BrandPost How TCS pioneered the ‘borderless workspace’ with Microsoft 365 Microsoft’s modern workplace solution proved a perfect fit for improving productivity and collaboration, while maintaining security of systems and data. By Tata Consultancy Services Mar 31, 2023 1 min Financial Services Industry Microsoft Cloud Computing BrandPost Supply chain decarbonization: The missing link to net zero By improving the quality of global supply chain data, enterprises can better measure their true carbon footprint and make progress toward a net-zero business ecosystem. By Tata Consultancy Services Mar 31, 2023 2 mins Retail Industry Supply Chain Green IT Podcasts Videos Resources Events SUBSCRIBE TO OUR NEWSLETTER From our editors straight to your inbox Get started by entering your email address below. Please enter a valid email address Subscribe