Challenging IT Analysts by Shifting Research From Vendors to Clients

A few months ago I wrote about a new company called Ombud and asked, can a new analyst firm take down Gartner?

By using a social media model, Ombud would, in effect, be a fourth-generation analysis firm. Analysis would come largely from volunteers, making it, in a way, the Huffington Post of analysis firms. (Disclosure: I now have a non-paid position on the Ombud board.

Ombud focuses more on the client than it does the vendor, product or solution. This idea seems so obvious in hindsight that I’m surprised we didn’t see it sooner. We all got so myopically focused that we didn’t step back and realize, “We are looking at this all wrong.”

The First Three Generations of IT Analyst Firms

As I mentioned in the last piece, we’ve been through three iterations of IT analyst firms. The first generation was all about numbers. IDC remains the strongest example of this set. While the numbers were mostly used by technology vendors, buyers could tell which products and platforms most of their peers were using and, through this, pick the ones that presented the least risk.

Latest From IDC: Big Data Hype Still Here, But Maturity Beckons

Gideon Gartner created the second and third generations of firms. First, Gartner Group took a model created to independently conduct stock analysis and adopted it to technology. He primarily targeted IBM, which dominated the market at that time.

When that model had aged, he came up with the third-generation analyst firm, Giga Information Group. Giga was heavily staffed by IT practitioners (including me) and ex-CIOs. We separated ourselves from vendor influence and revenue to better assure independence. Unfortunately, for reasons ranging from uneven management to a disloyal board, the company didn’t survive.

The problem with both the second- and third-generation firms is that they started with a model based on financial analysts, who have virtually nothing in common with those who buy IT solutions. The next generation has to correct that mistake.

Overcoming the Bias of User Reviews

The fourth-generation IT analyst firm is being built as a social exercise. Ombud is one of several firms vying for the position. Initially, it was believed that some blend of Yelp and Facebook would be ideal—largely because we all got so darn excited about social media. Then Facebook faceplanted its IPO, while we began to understand that Yelp reviews were uneven in quality.

News: Yelp Cracks Down on Fake Reviews

A Yelp-like problem emerged: You couldn’t really trust reviews being generated socially. There were three reasons why:

1. Both a vendor and its competitors had a nasty tendency to try to game the system by posting false positive or negative reviews.
2. Reviewers didn’t actually know how to write a good review, as it wasn’t really their job and they often didn’t have the time or inclination to do one.
3. Even an independent, well-written review might not apply to any other situation.

Now, you could do background checks and vet the reviewers to limit vendor games, but it’s nearly impossible to eliminate them. You could provide incentives to submit thorough, well-written reviews, too. It’s the third point—assuring the review actually applies to the situation—that’s much tougher.

For instance, vendors often provide as advocates IT managers who have purchased their products. I remember when Microsoft had an event called Scalability Day to show that its Windows NT server could replace a mainframe. I called the IT manager “advocate.” He confirmed that Windows NT servers had successfully replaced his mainframes, but he also said he could have bought three mainframes for what this Microsoft-funded deployment to replace one had actually cost. Vendor-funded early adopters are common, but it’s important to note that IT shops are very different.

Commentary: Oracle’s Advertising Claims Are Strategically Stupid

Old style or new, unless you can assure the review, selection, implementation and quality assurance processes, the best the analyst firm can do is provide cover-your-butt documentation that makes it look like you did a good job. In fact, that’s what the existing firms are mostly used for today. While that has a ton of value, actually assuring the result would be better.

How the Fourth Generation Assures the Result

While Ombud continues to have socially reviewed content, it has shifted to a model that focuses on a consistent and tested vendor and product selection process, rather than a vendor or product answer. The power of the social network shifts from outside to inside resources, while engagement is monitored and reported from cradle to grave. This clearly works best with large enterprises; you need a critical mass of folks inside the company, engaged with the vendors and their product sets, to provide feedback.

In this manner, the vendor and solution evaluation, rather than being done in a vacuum or by someone in an unrelated firm, is done by a team that consists largely of internal resources and is managed against a tested process.

When a firm comes to Ombud, here’s what happens:

• Ombud and its network of IT executives define a pool of products for consideration and then help the firm create selection criteria to measure the solutions.
• Ombud goes to the vendors on behalf of the company. Vendors may or may not know who the company is, which potentially limits internal lobbying efforts.
• Ombud collects and scores the responses based on client requirement and gives the analysis and result to the client for review and feedback. (This last step assures that the result is consistent with both stated and unstated requirements and to improve the process over time.)
• Finally, Ombud measures vendor execution against the requirements. Over time, Ombud gains perspective on the vendors’ relative ability to meet requirements and the teams the vendor deploys.

The New Analyst Normal: Know Your Enemy

From the beginning, IT analyst firms were designed to assure the solution selection process. Some did get involved in procurement, but mostly as an afterthought. It amazes me that it took us this long to realize that the best result might come from focusing on assuring the procurement process itself because, done right, that would better assure the solution choice.

After going through this exercise, it surprises me how much we depend on traditional reviews. Whether choosing cars, movies, appliances or mates, we rarely agree with professional reviewers, because their requirements don’t match our own. We’ve all made mistakes, which suggests that our own review processes need work.

We’re seeing the beginning of the fourth-generation IT research firm, and it reflects a shift in research focus from the solution to the client. This is basic Sun Tzu. Since I’m a fan, I’m surprised I didn’t see it sooner.

Rob Enderle is president and principal analyst of the Enderle Group. Previously, he was the Senior Research Fellow for Forrester Research and the Giga Information Group. Prior to that he worked for IBM and held positions in Internal Audit, Competitive Analysis, Marketing, Finance and Security. Currently, Enderle writes on emerging technology, security and Linux for a variety of publications and appears on national news TV shows that include CNBC, FOX, Bloomberg and NPR.

Follow everything from CIO.com on Twitter @CIOonline, Facebook, Google + and LinkedIn.