Reference material

As part of our regular research we publish annual reviews of the main technology markets that we cover, and as part of this we take up customer references from software vendors.

When I designed our methodology for assessing vendors I was rather concerned that taking up customer references would not be particularly insightful.

I thought it would be like taking up personal references for an employee applying for a job, which rarely yields a great deal of information — how hard should it be for someone to find a few ex-colleagues to say something nice about them?

I was concerned that, by sending out satisfaction surveys to reference customers of a vendor, I would just end up with a series of perfect satisfaction scores and positive comments in every case, which would not be much use in discriminating between vendors.

How wrong I was.

Every time we have been through this process it has yielded a number of insights. The first is the realisation of just how far removed many vendors are from their customer implementations.

They sell the software and possibly do some consulting to help implementation, but beyond that only the vendor’s technical helpdesk has much customer contact, apart from an account manager who calls up once a year to collect the maintenance fee and perhaps some contact at user group meetings.

In many cases the vendor may have little idea how their software is perceived at the customer site, particularly if the software is deployed on multiple projects.

From a vendor viewpoint I would have expected vendors to have cultivated deeper relationships with some customers since they need their references in order to help sell to new customers.

Happy customers are an important resource to vendors, who will use them with prospective customers, venture capitalists considering a new investment round and indeed with pesky analysts such as myself.

What we have found is that there is a wide gulf between vendors when it comes to references. We ask firms for a minimum number of references but give credit to a vendor who supplies more, as this indicates a level of confidence: they are not just cherry-picking their happiest customers but are happy to be assessed on a broader base of clients.

Vendor size does not appear to be a factor, and some small vendors send us large numbers of references while larger ones struggle to provide the minimum we require.

We then ask the references how happy they are with the software, the company and various aspects of the vendor experience, such as support and billing.

We ask whether the software works as advertised, and whether the customer has achieved the benefits they had hoped for.

We also ask the customer to comment on the best and worst aspects of the software, and to answer what I initially worried was a superfluous question: “Would you recommend this software to others?” Given that these references are supplied by the vendor, one might assume that the answer to this final question would always be “Yes”.

Remarkably, in every cycle we get some supposed references saying they would not recommend the software to others, usually explaining at length the difficulties they have had with it.

I find this extraordinary. What’s more, the satisfaction scores vary dramatically from vendor to vendor, with some barely able to find a really happy customer and others having a high proportion of very happy clients.

Interestingly, some of the vendors who supply large numbers of references often have very high average feedback scores, while others who have carefully selected their happiest customers struggle to provide references that are more than merely satisfied.

We also tell the vendors where their overall feedback scores rank relative to the peer group, and in our reports we credit the top five vendors with the happiest customers.

The vendors who come at the bottom of the heap, interestingly, are often the ones who show little concern about the fact.

These are often the same vendors who have been aggressively pointing out to me in briefings just how much better their products are than the competition.

They seem either blissfully unaware or are simply unconcerned that many of their own reference customers are barely satisfied with their software.

Perhaps this is a function of the sell-and-move-on mentality of many enterprise software vendors, which are driven by quarterly sales figures to the exclusion of everything else. It should surely be of concern to buyers, who often seem influenced by brand rather than carrying out proper due diligence when purchasing software.

My advice to prospective purchasers is to insist on multiple customer references when purchasing software, and to invest the time in checking just how successful those customers have actually been.

A little digging around before purchasing may save a lot of suffering later.

Andy Hayler is founder of research company The Information Difference. Previously, he founded data management firm Kalido after commercialising an in-house project at Shell