When Vendor Research Collides: Whom to Believe?

There are times in the high-tech world when it’s particularly vexing to figure out just who to believe.

Similar survey data (ostensibly from the same general survey base) can provide contradictory results. Warring vendors bend facts and figures to suit their purposes and unleash FUD. And statistics can be contorted to say just about anything.

Such is the case with two pieces of research and survey data regarding SAP’s Solution Manager, an application lifecycle management product that is unsexy as they come though critical to many SAP customers. Or so it would appear.

According to SAP, the Solution Manager “facilitates technical support for distributed systems—with functionality that covers all key aspects of solution deployment, operation and continuous improvement. A centralized, robust application management and administration solution, SAP Solution Manager combines tools, content, and direct access to SAP to increase the reliability of solutions and lower total cost of ownership.”

The overuse of the word solution notwithstanding, SAP’s “SolMan” product and its new functionalities got warm reviews from an IDC whitepaper published in late 2009. (IDC and CIO.com are both owned by the same parent company, IDG.) The document, by IDC’s Elaina Stergiades and Eric Hatcher, was based on interviews IDC conducted with seven SAP customers “to capture their experiences since implementing and using SAP Solution Manager,” notes the report.

The results that follow (taken directly from the report) were glowing:

• On average, SAP Solution Manager customers did not have to increase their IT staff sizes to support new activity. This results in an average savings of $393,307 because the customers avoided new hires. Additionally, the current IT staff is 11% more productive since the deployment.
• On average, downtime incidents were reduced by 8.4% and hours per incident were cut by 50.8%.
• The three-year discounted benefit is an average total of $4,489,438 or $126,378 per 1,000 users.
• The results show an aggregate ROI of 463%; the payback period is 4.3 months.
• Results show that customers using SAP Solution Manager in both small and larger numbers of scenarios achieve an ROI over 200% and payback in less than one year.

It probably won’t shock you to learn that SAP footed the bill for the research. Clearly written at the top of Page 1 is “Sponsored by: SAP.”

Now let’s look at a very different set of survey data on Solution Manager and its value to SAP customers, this time from a company named Panaya, which is not a market researcher.

Before I tell you what type of company Panaya is, let’s look at the results of their online survey of “347 SAP customers and systems integrators worldwide in April 2010 to learn how SAP customers use SAP Solution Manager and what challenges and benefits they associate with it,” notes a Panaya press release.

Not wanting to bury the lead, here’s the first sentence of the press release: “The complexity of SAP Solution Manager is its biggest challenge.”

The results that follow (taken directly from the Panaya report) were not as glowing, when compared to the IDC report:

• The top benefit of using SAP Solution Manger is Control (42%). A substantial 20% of the respondents are unsure of what benefit they receive from SAP Solution Manager.
• The Complexity of SAP Solution Manager is the biggest challenge in using the system (32%). One quarter of the respondents do not understand the value of SAP Solution Manager and find that to be a challenge.
• Documenting business process in SAP Solution Manager is key for using some of its advanced functionality. But 42% of the respondents have not documented any of their business processes in SAP Solution Manager and only 3% have fully documented their business processes.
• System Monitoring is the most used feature of SAP Solution Manager. Service Desk, Job Scheduling, and Incident Management are the least used features.
• On average, SAP Solution Manager features go unused by 51% of the respondents, 10% use them rarely, 16% use them partially, 12% use them quite a bit, and only 8% use them fully.

So what does Panaya do? It “provides software tools that save SAP customers up to 50% of their software upgrade and maintenance costs, while minimizing risks and proving clear ROI,” according to the company. Again, the survey was done by Panaya, and not a third-party research firm.

In sum: SAP sponsors research by IDC that provides positive reviews and eye-popping ROI regarding its Solution Manager product. And research done by Panaya, a company that aims to undercut and displace SAP’s software support and upgrade costs, discovers too much complexity, unrealized benefits and use-case challenges with Solution Manager.

What the Vendors Have to Say

In response to the Panaya research, SAP’s Jim Dever, head of the Americas media relations, writes in an e-mail: “Our own recent research demonstrates the significant returns on investment that can be achieved, and we are committed to providing tools and educational resources that help our customers better understand the benefits of Solution Manager and how to put it to best use within their IT landscapes.”

As to veracity of Panaya’s SolMan research, Dever points to the fact that Panaya did “not bother to engage with a reputable third party.”

Panaya CMO Amit Bendov counters in an e-mail that “we do not have any motive to bash Solution Manager—it is not a competitor. While we are certainly in the application management business and are reducing SAP support costs, the functional overlap with SolMan is marginal at best. In fact, most (and possibly all) of our customers have solution manager installed.”

Bendov insists that Panaya wanted market research on SolMan “for our own needs and could not find anything from SAP or others.”

So, whom do you believe? SAP? Panaya? Both? Neither?

Do you Tweet? Follow me on Twitter @twailgum. Follow everything from CIO.com on Twitter @CIOonline.