That’s because software providers routinely exaggerate their assessments, leading CIOs to believe that the entire company has been assessed at a certain level when only a small slice of the company was examined. And once providers have been assessed at a certain level, there is no requirement that they test themselves ever again?even if they change dramatically or grow much bigger than they were when they were first assessed. They can continue to claim their CMM level forever.

Worse, some simply lie and say they have a CMM assessment when they don’t. And appraisers say they occasionally hear about colleagues who have had their licenses revoked because of poor performance or outright cheating in making assessments.

Yet CIOs who want to check up on CMM rating claims are out of luck. There is no organization that verifies such claims. Furthermore, the Software Engineering Institute (SEI), which developed CMM and is principally funded by the DoD, will not release any information about companies that have been assessed, even though appraisers are required to file records of their final assessments with the institute.

As American and European companies stampede offshore to find companies to do their development work, they first need to understand what CMM ratings really mean. Yet few CIOs bother to ask crucial questions, say IT industry analysts and the service providers themselves. "Not even 10 percent of customers ask for the proof of our CMM," says V. Srinivisan, managing director and CEO of ICICI Infotech, an Indian software services provider that claims a Level 5 certification. "They inevitably take it for granted, and they don’t ask for the details."

CIOs who don’t ask for the details will not be able to distinguish between companies that are using CMM in the spirit it was intended?as a powerful, complex model for continuous internal improvement?and those that are simply going through the motions to qualify for business. Buying by the CMM number alone could mire CIOs in the same problems that caused them to look offshore in the first place: high costs, poor quality and shattered project timetables?not to mention the loss of thousands of U.S. IT jobs.

"When you talk about something simple like a number and lots of money is involved, someone’s going to cheat," says Watts Humphrey, the man who led the development of CMM and is currently a fellow at the SEI. "If CIOs don’t know enough to ask the right questions, they will get hornswoggled." (For a list of the best questions to ask, see "Twelve Critical Questions," Page 52.)