Indeed, CIOs who look to CMM for guarantees won’t find them, says Rick Harris, director of application development for OnStar, a division of GM that provides communications inside the company’s vehicles. He recalls confronting a manager from one of his CMM Level 5 offshore outsourcing companies who did not know how to do a testing plan for software. "My people had to train him to do it," he says. On another occasion, Harris’s staff discovered that the offshore provider had fallen far behind schedule in one of its projects but had not told him. "You’d think a Level 5 company would have told me months before that the schedule was slipping and we needed to do something," he says.

Problems like those can damage CIOs’ credibility inside IT and with the business?especially if they used a CMM level to defend a decision to move development offshore or use a particular outfit. As Harris has learned, what matters is what’s behind the impressive-looking number. Is there a verifiable commitment to quality, process and training? Can companies demonstrate improvements they’ve made over time in customer delivery times, developer productivity and defect density? Will the project managers that went through the assessment be assigned to your project? If the answer to any of these questions is no, then a CMM Level 5 isn’t worth much.

There is still no substitute for deep due diligence. "The real test is when you get into the trenches and see whether these companies bring their capabilities to bear," says Harris. "Do their people and processes hold up under pressure? In my experience, in some cases they have and others they haven’t."