There are times in the high-tech world when it's particularly vexing to figure out just who to believe. \n\nSimilar survey data (ostensibly from the same general survey base) can provide contradictory results. Warring vendors bend facts and figures to suit their purposes and unleash FUD. And statistics can be contorted to say just about anything. \n\nSuch is the case with two pieces of research and survey data regarding SAP's Solution Manager, an application lifecycle management product that is unsexy as they come though critical to many SAP customers. Or so it would appear. \n\nAccording to SAP, the Solution Manager "facilitates technical support for distributed systems\u2014with functionality that covers all key aspects of solution deployment, operation and continuous improvement. A centralized, robust application management and administration solution, SAP Solution Manager combines tools, content, and direct access to SAP to increase the reliability of solutions and lower total cost of ownership." \n\nThe overuse of the word solution notwithstanding, SAP's "SolMan" product and its new functionalities got warm reviews from an IDC whitepaper published in late 2009. (IDC and CIO.com are both owned by the same parent company, IDG.) The document, by IDC's Elaina Stergiades and Eric Hatcher, was based on interviews IDC conducted with seven SAP customers "to capture their experiences since implementing and using SAP Solution Manager," notes the report. \n\nThe results that follow (taken directly from the report) were glowing: \nOn average, SAP Solution Manager customers did not have to increase their IT staff sizes to support new activity. This results in an average savings of $393,307 because the customers avoided new hires. Additionally, the current IT staff is 11% more productive since the deployment.\nOn average, downtime incidents were reduced by 8.4% and hours per incident were cut by 50.8%.\nThe three-year discounted benefit is an average total of $4,489,438 or $126,378 per 1,000 users.\nThe results show an aggregate ROI of 463%; the payback period is 4.3 months.\nResults show that customers using SAP Solution Manager in both small and larger numbers of scenarios achieve an ROI over 200% and payback in less than one year.\nIt probably won't shock you to learn that SAP footed the bill for the research. Clearly written at the top of Page 1 is "Sponsored by: SAP." \n\nNow let's look at a very different set of survey data on Solution Manager and its value to SAP customers, this time from a company named Panaya, which is not a market researcher. \n\nBefore I tell you what type of company Panaya is, let's look at the results of their online survey of "347 SAP customers and systems integrators worldwide in April 2010 to learn how SAP customers use SAP Solution Manager and what challenges and benefits they associate with it," notes a Panaya press release. \n\nNot wanting to bury the lead, here's the first sentence of the press release: "The complexity of SAP Solution Manager is its biggest challenge." \n\nThe results that follow (taken directly from the Panaya report) were not as glowing, when compared to the IDC report: \nThe top benefit of using SAP Solution Manger is Control (42%). A substantial 20% of the respondents are unsure of what benefit they receive from SAP Solution Manager.\nThe Complexity of SAP Solution Manager is the biggest challenge in using the system (32%). One quarter of the respondents do not understand the value of SAP Solution Manager and find that to be a challenge.\nDocumenting business process in SAP Solution Manager is key for using some of its advanced functionality. But 42% of the respondents have not documented any of their business processes in SAP Solution Manager and only 3% have fully documented their business processes.\nSystem Monitoring is the most used feature of SAP Solution Manager. Service Desk, Job Scheduling, and Incident Management are the least used features.\nOn average, SAP Solution Manager features go unused by 51% of the respondents, 10% use them rarely, 16% use them partially, 12% use them quite a bit, and only 8% use them fully.\nSo what does Panaya do? It "provides software tools that save SAP customers up to 50% of their software upgrade and maintenance costs, while minimizing risks and proving clear ROI," according to the company. Again, the survey was done by Panaya, and not a third-party research firm. \n\nIn sum: SAP sponsors research by IDC that provides positive reviews and eye-popping ROI regarding its Solution Manager product. And research done by Panaya, a company that aims to undercut and displace SAP's software support and upgrade costs, discovers too much complexity, unrealized benefits and use-case challenges with Solution Manager. What the Vendors Have to SayIn response to the Panaya research, SAP's Jim Dever, head of the Americas media relations, writes in an e-mail: "Our own recent research demonstrates the significant returns on investment that can be achieved, and we are committed to providing tools and educational resources that help our customers better understand the benefits of Solution Manager and how to put it to best use within their IT landscapes." \n\nAs to veracity of Panaya's SolMan research, Dever points to the fact that Panaya did "not bother to engage with a reputable third party." \n\nPanaya CMO Amit Bendov counters in an e-mail that "we do not have any motive to bash Solution Manager\u2014it is not a competitor. While we are certainly in the application management business and are reducing SAP support costs, the functional overlap with SolMan is marginal at best. In fact, most (and possibly all) of our customers have solution manager installed." \n\nBendov insists that Panaya wanted market research on SolMan "for our own needs and could not find anything from SAP or others." \n\nSo, whom do you believe? SAP? Panaya? Both? Neither? \n\n Do you Tweet? Follow me on Twitter @twailgum. Follow everything from CIO.com on Twitter @CIOonline.