Just as customer satisfaction is a key goal for companies, ensuring end users are satisfied is an important responsibility of CIOs. In CIO’s "State of the CIO 2003" survey, 63 percent of respondents said they regularly measure either internal or external customers’ satisfaction with IT’s services. But only 36 percent said this is a highly effective practice for adding value to their business. The discrepancy isn’t surprising. My experience with customer satisfaction measurement in the technology industry suggests that companies?and by extension CIOs?often talk to the wrong people in the customer organization and measure the wrong things. If you aren’t careful, customer satisfaction measurement could be doing your IT organization more harm than good. Here are some tips for getting customer satisfaction measurement right.

Measure Success, Not Satisfaction

Consider the experience of Trilogy, one of the largest privately held software companies. Trilogy employed a classic multivariable customer satisfaction measurement system. It contracted respected outside companies to hone a 40-point index addressing all major drivers of customer satisfaction. Semiannually, these companies would survey a cross-section of customers and analyze and report this data to management. Trilogy management took the results very seriously. Then one day, only a week after Trilogy executives were congratulating themselves on a key customer’s high satisfaction ratings, a senior executive from that account threatened to shut down its project.

What went wrong? Trilogy found that its surveys focused on many intermediate metrics, such as "Do you like our people?" or "Are we easy to work with?" But these measures shed little light on the end result: Was the customer’s experience with the software project a success? Trilogy realized that customers do not care about the technical merits of the software or the responsiveness of the account manager. Customers initiate projects to drive specific operational and financial changes in their businesses. The only thing they care about is the actual business value delivered by the vendor. Trilogy now measures customer success on a set of "business success metrics" established jointly with each customer. These metrics are relevant to the senior business sponsor and are designed to measure the value delivered throughout the life of the project.

Don’t Let Averages Lie

Sometimes, it’s not obvious who the customer really is. Enterprise technology projects involve multiple constituents with differing and often conflicting objectives. IT investment decisions involve business decision-makers, IT professionals, finance executives and end users. These audiences may define value differently.

For instance, in deploying a CRM system, the IT organization may care about ease of deployment and performance. The finance organization may emphasize ROI. The end users may care about ease of use and adaptability of the software to their needs. Given these differences in priorities, one audience could be satisfied while another is unhappy with the same project. An average customer satisfaction score will completely mask these differences. In the Trilogy example, the IT organization was pleased with the vendor, but the business sponsor was dissatisfied. And because the business sponsor was paying the bills, his opinion was the one that mattered.