Like many CIOs, Steve Pickett is happy to serve as a reference for his favorite vendors. If he spends a couple of hours talking to a prospective customer, the vendor might not bill his company the next time he needs a service visit not covered in his contract. He doesn’t benefit personally from the deal; the perks go to his company, which approves of the practice, so why not?

But when the time comes for Pickett to make a decision about buying his own software, he doesn’t place much stake in what his colleagues have to say?and he doesn’t always think that the CIOs who call him should either.

"Some of the companies that have called me have taken [the reference process] much too lightly," says Pickett, vice president and CIO for Penske, a Detroit-based transportation services company. "If they’re calling me to find out whether a particular supplier is doing a good job, without knowing the

complexities within my company, I’m not sure that they’re learning anything. I may be giving them misleading information because their business is different from mine."

Pickett says he’s up front about this. Nevertheless, he admits there is a natural tendency to avoid the negatives if things are going well. "If you have a good relationship with the supplier, you’re not going to bad-mouth them," he says, though he’s careful to point out that if he’s receiving bad service from a vendor, he’ll say so.

All of which is to say, customer references come with baggage?big, unwieldy baggage. Vendors’ long-held practice of offering enticements to customers to speak to potential new customers is widespread and widely accepted by CIOs. Yet despite the fact that reference checks are possibly the most important step in the software selection process, CIOs generally know little about the references they call and even less about the terms of the arrangement between the reference and the vendor.

Companies that serve as references are likely to get things that average customers never get: preferential treatment, influence on the development cycle?even cash rebates. These customers aren’t average in any way and might not reflect the reality of your situation unless you become a reference too. And even if the reference isn’t getting special treatment, most software projects have become so big and complex that no two companies are likely to have the same experience anyway. All this means that customer references?especially in the realm of big enterprise software projects like ERP and CRM?have ceased to have much real meaning.