David Constant, a lead appraiser and partner with Process Inc., a software projects consultancy, recalls assessing a company where all the developers had been coached by management on what to say. "I had to stop the interviews and demand to see people on an ad hoc basis, telling the company who I wanted to speak to just before each interview began," Constant recalls. "And the sad part was that they didn’t need to coach anybody. They would have easily gotten the level they were looking for anyway?they were very good."

The new model is much tougher to exploit than the original questionnaire. In 1991, Westinghouse’s Martak recalls telling his management: "This is a different ball game now. If you have a good lead appraiser, you can’t fake it out." Martak led his group to a Level 4 assessment and eventually became a lead appraiser himself.

The depth and wisdom of the CMM itself is unquestioned by experts on software development. If companies truly adopt it and move up the ladder of levels, they will get better at serving their customers over time, according to anecdotal evidence. But a high CMM level is not a guarantee of quality or performance?only process. It means that the company has created processes for monitoring and managing software development that companies lower on the CMM scale do not have. But it does not necessarily mean those companies are using the processes well.

"Having a higher maturity level significantly reduces the risk over hiring a [company with a lower level], but it does not guarantee anything," says Jay Douglass, director of business development at the SEI. "You can be a Level 5 organization that produces software that might be garbage."

That assessment is borne out by a recent survey of 89 different software applications by Reasoning, an automated software inspection company, which on average found no difference in the number of code defects in software from companies that identified themselves on one of the CMM levels and those that did not. In fact, the study found that Level 5 companies on average had higher defect rates than anyone else. But Reasoning did see a difference when it sent the code back to the developers for repairs and then tested it again. The second time around, the code from CMM companies improved, while the code from the non-CMM companies showed no improvement.

Truth in Advertising

Stories about false claims abound. Ron Radice, a longtime lead appraiser and former official with the SEI, worked with a Chicago company that was duped in 2003 by an offshore service provider that falsely claimed to have a CMM rating. "They said they were Level 4, but in fact they had never been assessed," says Radice, who declined to name the guilty provider.