Like many CIOs, Steve Pickett is happy to serve as a reference for his favorite vendors. If he spends a couple of hours talking to a prospective customer, the vendor might not bill his company the next time he needs a service visit not covered in his contract. He doesn\u2019t benefit personally from the deal; the perks go to his company, which approves of the practice, so why not? But when the time comes for Pickett to make a decision about buying his own software, he doesn\u2019t place much stake in what his colleagues have to say?and he doesn\u2019t always think that the CIOs who call him should either."Some of the companies that have called me have taken [the reference process] much too lightly," says Pickett, vice president and CIO for Penske, a Detroit-based transportation services company. "If they\u2019re calling me to find out whether a particular supplier is doing a good job, without knowing thecomplexities within my company, I\u2019m not sure that they\u2019re learning anything. I may be giving them misleading information because their business is different from mine."Pickett says he\u2019s up front about this. Nevertheless, he admits there is a natural tendency to avoid the negatives if things are going well. "If you have a good relationship with the supplier, you\u2019re not going to bad-mouth them," he says, though he\u2019s careful to point out that if he\u2019s receiving bad service from a vendor, he\u2019ll say so.All of which is to say, customer references come with baggage?big, unwieldy baggage. Vendors\u2019 long-held practice of offering enticements to customers to speak to potential new customers is widespread and widely accepted by CIOs. Yet despite the fact that reference checks are possibly the most important step in the software selection process, CIOs generally know little about the references they call and even less about the terms of the arrangement between the reference and the vendor.Companies that serve as references are likely to get things that average customers never get: preferential treatment, influence on the development cycle?even cash rebates. These customers aren\u2019t average in any way and might not reflect the reality of your situation unless you become a reference too. And even if the reference isn\u2019t getting special treatment, most software projects have become so big and complex that no two companies are likely to have the same experience anyway. All this means that customer references?especially in the realm of big enterprise software projects like ERP and CRM?have ceased to have much real meaning.Worse, serving as a reference yourself can have tragic consequences for your company and your career. Accepting personal gifts or money for being a reference is unethical and can, of course, get you fired. But even if the gifts and special treatment benefit only your enterprise, not you, they can still violate your company\u2019s code of ethics or jeopardize your company\u2019s fortunes on Wall Street. Saying good things about bad or problematic software can send up a red flag with Wall Street analysts, who now track software projects more closely and downgrade company stocks at the slightest whiff of trouble. The reference game is enough of an ethical quagmire that the federal government has banned its IT executives from making endorsements of any kind.The government may be on to something. The benefits that companies gain from acting as references for vendors may no longer justify the risks?both personal and companywide?that are part of the package. In fact, the only viable reason for agreeing to be a reference for a supplier may turn out to be the simplest one of all: because you think it\u2019s good.The Care and Feeding of YouAccording to one survey by the IT Services Marketing Association, 52 percent of IT buyers make purchase decisions based on referrals from colleagues. If you don\u2019t have a buddy you can call for a reference, vendors are ready to help. They enlist and groom armies of happy reference customers to take calls about their experiences. "There\u2019s a science to the care and feeding of these references," says Analyst Jim Shepherd of Boston-based AMR Research. That\u2019s because references are the cornerstone of the software sales cycle.Being a reference can be informal and simple?such as agreeing to be listed on the vendor\u2019s website or doing the occasional favor for the vendor\u2019s sales team. Or it can be quite complex, with specific agreements written into the software contract between the reference and the vendor. The contract might stipulate taking a certain number of calls from prospective customers, the press or analysts, for example, or speaking at a user conference, or approving a certain number of press releases and case studies about the implementation.At large software companies like Pleasanton, Calif.-based PeopleSoft, managing these references is a full-time job. In fact, it\u2019s five full-time jobs, overseen by Wendi Wolfgram, program manager for the user reference program, for which more than 700 companies have signed up. Wolfgram and her staff keep a database of information about those customers, including details about each installation, multiple contacts in different parts of the business, and the customer\u2019s level of commitment to and history of serving as a reference. When a salesperson needs a reference for a prospective customer, the reference team looks for a good match?a company in a similar industry that\u2019s not a direct competitor, that has a similar shop and that hasn\u2019t been overused as a reference. According to Wolfgram, the program is so valuable that it affects more than 78 percent of deals closed each quarter. And it\u2019s so valuable that PeopleSoft puts a great deal of time, effort and money into keeping it running smoothly."We do have rewards, and it\u2019s kind of like a frequent flier program," says Heather Loisel, PeopleSoft\u2019s vice president of global sales productivity. "It\u2019s completely voluntary, and it kicks in after a certain level of volunteer activity. The reference earns points, and those points can be used for things like a pass to one of our member conferences."Givers and TakersFree conference passes are only the beginning. "You usually get something out of it," says Eastman Kodak CIO Mark Gulling, who has gotten free consulting and, from SAP, a rebate for serving as a reference. He won\u2019t give specifics but says the rebate is minimal and benefits his Rochester, N.Y.-based employer, not him. "[Vendors] understand that it takes time for you to do this, but that\u2019s not why you do it," he says. "It gives me contacts that I can go back and talk to later."The vendors CIO spoke with for this story?i2 Technologies, Manugistics, Oracle and PeopleSoft?all downplayed these incentives. (Other competing vendors declined to be interviewed for this article.) "We don\u2019t want to have a program in place where the reward is so large that the customer says, I\u2019m going to go after that award," says Katrina Roche, chief marketing officer for Dallas-based i2 Technologies. "We want people to know we appreciate them, but we don\u2019t want that appreciation to sway that decision."But it would be naive to assume that shady reference deals aren\u2019t cut in the software industry. Roche remembers working for one small software vendor that offered a customer a $7,000 discount to serve as a "marketing center"?participating in ad campaigns and the like.Dave Munn, CEO of Lexington, Mass.-based IT Services Marketing Association, says typical reference deals involve credit toward training classes or a future purchase. And a check in the mail? "Clearly that happens," he says.Patti Morrison, CIO of Office Depot in Delray Beach, Fla., and former CIO of GE Industrial Systems and Quaker Oats, does not count herself among the takers. But she does feel pressure from vendors that want her to put a reference agreement in the software contract?especially smaller startups trying to solidify their position in the marketplace with references so that they can get financing. "They say, If you do a press release by the end of our first quarter, we\u2019ll give you X percent discount," Morrison says. "They might use it as a negotiating tactic for price, but that [discount] is not a good decision on which to make the choice of that vendor." Morrison, whose company has a policy of not putting reference agreements into contracts, tells them that she\u2019ll serve as an informal reference?only after the project is complete and has been successful. But even in this reduced role she still benefits, albeit in a roundabout way."It\u2019s an additional incentive for the vendor to support you," she says, noting that serving as a reference sometimes gets her better customer service, including access to early software releases, and better response from the vendor\u2019s senior managers and software developers. "It\u2019s a great way to build a relationship with the supplier [and get] their skin in the game to make you successful." Blinded by the LighthouseAmong companies that agree to link their name to a vendor, some have more value as references than others. These are the giants, the companies people instantly recognize, the brands they trust. A short jaunt through any vendor\u2019s annual report, conference lineup or website makes it clear which customers are so-called lighthouse accounts: Procter & Gamble and Nestl\u017d for SAP; Charles Schwab and Deutsche Telekom for Siebel; Dell and Home Depot for i2 Technologies.These lighthouses can help vendors gain legitimacy, sell products and boost shareholder confidence by proving that the vendor has a steady paycheck from a company with deep, well-constructed pockets. But they can also break their vendors, by talking up a project and then pulling back on it. Last February, for example, TheStreet.com reported that Manugistics\u2019 stock price fell 18.5 percent reportedly after a research analyst said that one of Manugistics\u2019 lighthouse customers, Ford Motor, had put some IT projects on hold. Manugistics\u2019 CFO Raj Rajaji had to assure nervous investors that Lloyd Hansen, Ford\u2019s vice president of revenue management, was speaking at an upcoming user conference?proof that the partnership was still strong.How Happy Talk WorksA reference may sound good, but it\u2019s rarely an accurate representation of reality. Companies that agree to serve as references are self-selected, happy customers, with a vested interest in the vendor leading a long, productive life. "You want your vendor to be successful and gain resources and clout so that they can improve their software," says Wendy Close, an analyst for Stamford, Conn.-based Gartner. This is one incentive for a CIO to become a reference?but it\u2019s also an incentive for a CIO to be a positive reference.References also use their position to influence the direction of the technology. "If I do things for [vendors], I end up having some influence on them in terms of the future direction I\u2019d like them to take," says the CIO of a retail company who spoke on condition of anonymity. When asked if that makes him less valuable as a reference, there\u2019s a long pause. "An average customer would have to think that customers like me have their best interest in mind," he says finally, noting that most prospects don\u2019t ask about his relationship with the vendor.It\u2019s worth taking the time to understand that relationship. In some cases, a company may form such a strong bond with a vendor that the company becomes something more than a customer if something less than a partner. But you have to dig to find that out for yourself?vendors won\u2019t spell it out for you. Consider, for example, Charles Schwab\u2019s affiliation with Siebel. The brokerage has long been prominent in Siebel\u2019s marketing materials. And a casual visitor to its website would assume that Schwab is nothing more than a good customer. But the CEO of Schwab has been on Siebel\u2019s board of directors since 1994, and back in 1996 the brokerage accounted for fully 23 percent of Siebel\u2019s revenue, according to an SEC filing. In other words, assuming that Schwab\u2019s experiences with Siebel will predict yours would require an imaginative leap.What Is Your Word Worth?Of course, there\u2019s nothing inherently wrong with forming a strategic partnership with a trusted vendor. And there\u2019s not necessarily anything wrong with accepting something in return for providing a reference either. But companies had better be willing to talk about it, ethics experts say."If a customer is unwilling to be transparent [about the rewards], then the conclusion that a disinterested party might draw is that the person has been bought," says Daryl Koehn, director of the Center for Business Ethics at the University of St. Thomas in Houston.So what can CIOs ethically accept in return for serving as a reference? Ideally, nothing, says Koehn. "If the vendor\u2019s products are good, the customer should be willing to make a recommendation for that vendor free of charge, right?" she asks. "The question is, Why does the customer feel entitled to demand some kind of recompense, or why does the vendor feel the need to offer some kind of recompense?" If the reference process doesn\u2019t take much time, the reference shouldn\u2019t take anything in return, Koehn suggests. But "if you\u2019re talking about the company\u2019s employees being tied up for days, it\u2019s not necessarily inappropriate for the customer to get something in return, like a small discount" or free training that takes up roughly the same time as the site visit, she says.Regardless, the CIO had better believe in the product without reservation. That\u2019s why some companies set strict parameters around referencing. For instance, The Hartford Financial Services Group has a policy forbidding its IT group from negotiating better prices or accepting free services in return for acting as a reference. "We pay for the technology and the services that we use in dollars, not in references, not in endorsements," says a spokesperson for the $15 billion insurance company in Hartford, Conn. "If we\u2019ve had a good experience with a vendor, we will occasionally share what the results were, but we try to make sure that nothing could get misinterpreted," says Hartford CIO David Annis. That means making sure that comments made during the reference process could not be interpreted as an endorsement.The federal government has barred its employees from endorsing vendors and accepting any gifts unless they go through an elaborate legal approval process. "A government employee who holds the decision-making authority cannot personally benefit from that authority," says Mark Forman, associate director for IT and e-government for the U.S. Office of Management and Budget. What\u2019s more, he says, the department as a whole cannot benefit from that kind of arrangement either. "I would consider that inappropriate," says Forman, the federal government\u2019s de facto CIO.Getting at the TruthIf the reference process is so troublesome, then how can CIOs make sure they\u2019re not getting hoodwinked? You can protect yourself; first, put the software through its paces as much as possible. Then, look for a trusted peer to talk to you about the software before you go to the vendor. That is why it\u2019s so important to stay in touch with your peers, either by attending conferences or joining a professional organization. "It\u2019s your personal relationship with the CIO that you have to leverage to find out what really happened. You won\u2019t get that out of a formal reference call," says the anonymous CIO.When he does talk to vendor references, this CIO asks for contacts at specific companies on the customer list rather than accepting the ones the vendors try to give him. When you call the phone numbers, ask the companies what they did to get on the list. "It\u2019s entirely fair to ask, What have you received from the vendor in exchange for allowing me to sit here in your conference room?" says Andrew McAfee, an assistant professor in technology and operations management at the Harvard Business School, who is doing research on the CIO decision-making process. "I would imagine it\u2019s a question that gets asked less often than it should."Merely knowing there are references is not enough. Rob Cohen, vice president and CIO at AstraZeneca, the London-based pharmaceutical company, says that two-thirds of the time, when he agrees to talk to a prospect, the company never calls. This suggests that some companies skimp on the homework?and that\u2019s dangerous. "I can\u2019t tell you the number of times where [vendors have] trotted out some very big name, but the reference is only using part of the product in one department," says Jim Sutter, senior partner at The Peer Consulting Group in Newport Beach, Calif., and former CIO of Xerox and Rockwell International.Companies that decide to do a site visit should send in people who will be on the front lines of the implementation, not just the CIO, Sutter recommends. "Particularly to a peer, [the CIO reference] will be reluctant to say, I really messed up when I made this decision," he says. "If you send some technical people in, they will uncover what worked and what hasn\u2019t worked."Then, during conversations, ask lots of questions?carefully, to extract facts, rather than opinions or generalities. For instance, you might ask, "How long does it take for them to get back to you? Not just, How\u2019s their service?" AstraZeneca\u2019s Cohen says.And remember Pickett, the reference who wouldn\u2019t listen to himself if he were making a purchase decision? That\u2019s not to say he finds references useless. He just won\u2019t use them to make a decision. Pickett has found that they\u2019re really only helpful once he\u2019s made up his mind."If it\u2019s a highly valued customer, I\u2019m going to find out from them what extra mile [the vendor was] willing to go to get [them]," Pickett says. He\u2019ll probe to find out, for instance, what the reference was able to get over the standard vanilla contract. Then, he\u2019ll ask for tips on how to actually implement a product?not pick it. "It\u2019s free advice if you use it correctly," he says.And if you don\u2019t? Well, you probably won\u2019t be worse off for following your own mind. After all, that\u2019s what Pickett does.