Some Advice on Advice: How to Get the Most from Your Research Services

There have been many changes in the technology analyst world in the past few years, and the last big change—Gartner’s acquisition of META Group—has left many research consumers wondering how to best get the accurate and independent advice they need. Here are a few suggestions on how to get the most from your research services.

Reach out to your users. Before you renew any service, ask your users how valuable the information has been in the past year. Does the research provide insight into the future? Have the analysts helped your users make real decisions or justify a business case? A number of companies with whom we work formally survey their research users each year and use the results to make renewal decisions. If you don’t have time for a formal process, some quick outreach should help you separate the “nice to have” from the necessary and give you some ammunition for any renewal negotiations.

Try before you buy. Don’t buy a new service from any firm without first doing a sample inquiry. Remember, what you’re really buying is access to the analysts—and what matters is how the analysts can help you get the answers you need in the next 12 months, not on any projected research schedule.

Ask questions and interact with analysts before you sign a contract. Submit a tough inquiry to each firm on your short list. Ask them the same question. You might find dramatically different answers, revealing how well they understand your needs, how deep they’re willing to go, and how general or specific their reply is.

Trying before you buy can help you determine the depth and independence of the research firm as well. Here’s a good test: Pick a technology with which you are very familiar, then ask the research firm to provide all of their research on that technology from the past 18 to 24 months. Does the research provide any insight you didn’t already know? Does it seem like they really understand the issues? Were they ahead of the announcements with thoughtful insight or just reacting to press releases? Does it seem like the analyst has seen and used the technology? (You might be surprised how few have.)

Answers to tactical questions are great, but check out their strategic insight as well—past and future. Ask for their research covering their predictions for 2004 and 2005. Were they right? Were they relevant to your company? Do they seem to have a coordinated view of the technology landscape or did they just pull together various predictions from a group of analysts?

Get real about actionable advice. Probably the most overused phrase in the research business is “actionable advice.” Just about every firm claims to deliver it, but it’s often hard to really find it within a sea of research notes. Again, examine the past research. Is that research filled with endless market “definitions” and “thought leadership” points, or can you actually do something with the advice they’ve delivered? And don’t overlook your current firms. If you didn’t get at least one “I’ve got to do this before the end of the week” tip in the past year then you may want to reconsider your plans to renew that service.

Achieve balance. Look for a vendor who complements your existing services and information sources. This could mean adding a market- or industry-focused vendor to the mix, or choosing another generalist firm with a different spin. For example, while Gartner and Nucleus Research are both generalist firms covering the broad technology market, their approaches to conducting analysis are quite different. Gartner analysts formulate opinions based on conversations with IT staff and vendors while Nucleus Research derives conclusions based on case study analysis of actual deployments. Neither is bad, just different.

Often companies choose a mix of both general and product- or industry-specific research. Product-specific analysis, such as CRM or ERP, tends to have value for a short term, as tough initial purchasing decisions are made once. Once made, you may not need that level of ongoing expertise. And since what you buy this year may be radically different from what you buy next year, it’s important not to get locked into a long-term contract if you don’t need this level of ongoing detail. For example, those purchasing a CRM system today may benefit from product-specific analysis now and a generalist going forward.

Demand transparency. How did an analyst arrive at a particular conclusion? Don’t just take his word for it. Your company will likely have a lot riding on the accuracy of their predictions. Find out what went into the thought process.

A note of caution: Beware of analysis based exclusively on broad survey data. Large surveys tend to be outsourced to survey houses, where low-level staff beat the phones to find participants in their questionnaires. Unfortunately relegating surveys to junior employees often results in low-quality information or data that’s missing critical context, as survey houses just don’t have the subject matter expertise to probe into issues that may arise during the survey process. In addition, these surveys seldom reach the real decision maker and even more rarely reach the CIO, since few CIOs are actually willing to participate in such surveys.

Survey information should be augmented with deeper analysis, which may be focused on your industry, your upcoming technology purchase or simply on a specialty area, such as supply chains or laboratory software. One strategy might be to stick with two different kinds of generalists and choose an industry-specific vendor, or choose a generalist with an alternate method.

Whichever your approach, look for vendors who bring an unbiased temperament. You should be able to uncover any bias from reading the research, but a quantitative factual approach is imperative. For example, a case study approach, by definition, delivers only the facts, free of subjective opinions. This may involve analysts speaking directly with numerous individuals involved in a deployment, moving beyond basic statistical surveys to gain a fuller picture of the environment.

And does the analyst firm sell to both the vendors as well as end users? This isn’t a reason to rule the firm out, but it’s important to be careful about adding too much of that to your mix. If you have two opinion-based companies, both of whom sell to the same technology vendor, you’re not likely to get an additional point of view by adding a third firm to your portfolio.

Gain broad access to research. No longer are tech purchases the exclusive domain of IT. Line-of-business managers, CIOs and others want insight into how to beat their competition and what practices to adopt for maximum ROI. If you’re still buying per seat or per hour, you’re not getting the advantages you should, and it may be time to renegotiate your contract.

Ignore forced bundling. The old days of a sales rep bundling a suite of services off a menu are gone. There are better deals out there. Some vendors will still try to bundle add-on services, but if all you want is a specific service, most firms will readily discuss options with you. A red flag to watch for: sales reps from research firms that are quick to sell. The sales rep should understand your market and be able to match their services to your needs. Here’s a quick test: Ask the sales rep to name your competitors. If they don’t understand your market, how can they offer help?

Don’t wear vertical blinders. The advantage of analysts with vertical-specific expertise is that they likely know a lot about your market and what your competitors are doing. However, relying on only vertical-specific advice may mean you miss opportunities to adopt best practices from other industries. If you’re looking only at case studies and data in your market, you may be keeping up with rivals but missing the leading-edge insight you really need to get ahead. For instance, an insurance carrier may want to know the content management strategies of its niche peers to make sure it’s keeping up, but may find best practices from manufacturing firms show an innovative way to use the technology to greater competitive advantage.

Finally, when evaluating potential research firms or making renewal decisions, keep in mind that a good analyst will help you make decisions with confidence—and support your strategic decisions with clear data and advice. Getting advice from organizations with complementary approaches can help provide a sanity check on other viewpoints you receive. Checking references, vetting analysts before you sign the contract and getting test inquiries answered will help ensure you get the attention and support you need.

Ian Campbell is the president and CEO of Nucleus Research, Inc., a broad-based technology advisory firm formed in 2000 that takes an ROI case study approach to its analysis.