CMM is a "snapshot in time," says the SEI, and it encompasses only the projects that were assessed. Furthermore, if the snapshot was taken more than two years ago, most experts say, it will have yellowed so badly that the company is probably no longer at the same maturity level.

Now that CMM has become table stakes for billions worth of business, some believe that providers should bite the bullet and get all their projects assessed if they are going to claim "enterprise Level 5 CMM."

"If I were a CIO and a company was telling me their entire company was CMM 5, I’d want all the people on my project to have gone through the assessment," says Margo Visitacion, a Forrester Research analyst and former quality assurance manager at a software development company. "[The service providers] are getting millions in business from their CMM levels. Why shouldn’t they have all of their developers go through an assessment?"

How Much for That Certification?

Appraisers continue to cheat too, according to their colleagues. The pressure on appraisers, in fact, is higher than ever today, especially with offshore providers competing in the outsourcing market. Frank Koch, a lead appraiser with Process Strategies Inc., another software services consultancy, says some Chinese consulting companies he dealt with promised a certain CMM level to clients and then expected him to give it to them. "We don’t do work for certain [consultancies in China] because their motives are a whole lot less than wholesome," he says. "They’d say we’re sure [certain clients] are a Level 2 or 3 and that’s unreasonable, to say nothing of unethical. The term is called selling a rating."

Will Hayes, quality manager for the SEI Appraisal Program, would only acknowledge one recent case of an appraiser who had his license revoked by the SEI for improperly awarding a company a Level 4 assessment. However, it’s difficult for the SEI to know exactly how much cheating is going on because it does not monitor the claims that companies make about CMM.

"Are there organizations out there claiming Level 5 who have never submitted the information to the SEI? I’m sure that there are," says SEI’s Douglass. That’s little comfort to CIOs who would rather not discover a false CMM claim the hard way?by seeing their projects fail.

There is a way to prove whether the assessment was done, but it may be hard for CIOs to get the evidence. Appraisers are required to submit formal documentation of all their assessments to the SEI and to customers. Lead appraisers must write up something called a Final Findings Report that includes "areas for improvement" if the appraiser finds any (they usually do, even with Level 5 companies). But there is no requirement for the content or format in the reports to be consistent across appraisers or companies. Only the methods for arriving at the final number are consistent. According to one appraiser who asked not to be named, companies will often ask appraisers to "roll up" the detailed findings into shallow PowerPoint presentations that don’t give a very good picture of the company and its software development processes. "The purpose of the report is to tell companies where they need to improve?that’s the whole point of CMM," she says. "But they make us write these fluffernutters that can gloss over important details."