Software Quality: Bursting the CMM Hype

1 2 Page 2
Page 2 of 2

But as soon as you dig beneath the surface, the logic falls apart. The process may have changed completely since the assessment was performed. Indeed, Indian services companies in particular, where the most CMM Level 5 assessments have been reported, are growing so quickly?some adding as many as 50 to 60 new developers a week?that avoiding change is nearly impossible. The company also may have changed the types of work it does and perhaps acquired other companies along the way that were not assessed at any level. And if the company does not have an excellent training program for all its project managers and developers?so they can work at the same level as those in the projects that were assessed?the assessment means little.

CMM is a "snapshot in time," says the SEI, and it encompasses only the projects that were assessed. Furthermore, if the snapshot was taken more than two years ago, most experts say, it will have yellowed so badly that the company is probably no longer at the same maturity level.

Now that CMM has become table stakes for billions worth of business, some believe that providers should bite the bullet and get all their projects assessed if they are going to claim "enterprise Level 5 CMM."

"If I were a CIO and a company was telling me their entire company was CMM 5, I’d want all the people on my project to have gone through the assessment," says Margo Visitacion, a Forrester Research analyst and former quality assurance manager at a software development company. "[The service providers] are getting millions in business from their CMM levels. Why shouldn’t they have all of their developers go through an assessment?"

How Much for That Certification?

Appraisers continue to cheat too, according to their colleagues. The pressure on appraisers, in fact, is higher than ever today, especially with offshore providers competing in the outsourcing market. Frank Koch, a lead appraiser with Process Strategies Inc., another software services consultancy, says some Chinese consulting companies he dealt with promised a certain CMM level to clients and then expected him to give it to them. "We don’t do work for certain [consultancies in China] because their motives are a whole lot less than wholesome," he says. "They’d say we’re sure [certain clients] are a Level 2 or 3 and that’s unreasonable, to say nothing of unethical. The term is called selling a rating."

Will Hayes, quality manager for the SEI Appraisal Program, would only acknowledge one recent case of an appraiser who had his license revoked by the SEI for improperly awarding a company a Level 4 assessment. However, it’s difficult for the SEI to know exactly how much cheating is going on because it does not monitor the claims that companies make about CMM.

"Are there organizations out there claiming Level 5 who have never submitted the information to the SEI? I’m sure that there are," says SEI’s Douglass. That’s little comfort to CIOs who would rather not discover a false CMM claim the hard way?by seeing their projects fail.

There is a way to prove whether the assessment was done, but it may be hard for CIOs to get the evidence. Appraisers are required to submit formal documentation of all their assessments to the SEI and to customers. Lead appraisers must write up something called a Final Findings Report that includes "areas for improvement" if the appraiser finds any (they usually do, even with Level 5 companies). But there is no requirement for the content or format in the reports to be consistent across appraisers or companies. Only the methods for arriving at the final number are consistent. According to one appraiser who asked not to be named, companies will often ask appraisers to "roll up" the detailed findings into shallow PowerPoint presentations that don’t give a very good picture of the company and its software development processes. "The purpose of the report is to tell companies where they need to improve?that’s the whole point of CMM," she says. "But they make us write these fluffernutters that can gloss over important details."

The Final Findings Report is what company officials present internally to the big brass and to customers knowledgeable enough to ask for it. But there’s no obligation to do it. They can declare their CMM level without producing any evidence. They can even hire their own lead appraisers inside the company and assess their CMM capabilities themselves. They don’t have to hire a lead appraiser from the outside who might be under less pressure to give a good assessment. And they can characterize their CMM level any way they want in their marketing materials and press releases.

SEI officials say they are not in the business of controlling what companies say about their assessments. Nor will they reveal to the public which companies have been assessed or what the assessments consisted of. "We weren’t chartered to be policemen?we’re a research and development group," Hayes says.

Instead, the SEI exerts control through the relatively small lead appraiser community (approximately 220 are authorized to do CMM assessments). From the beginning, the SEI has reserved the right to discipline or even remove appraisers who cheat or do their jobs badly. But in the early days, the SEI rarely followed through on those threats, say longtime appraisers.

More recently, the SEI toughened up the CMM itself and plans to completely replace it (as of December 2005) with a broader, more in-depth model called CMMI. In the process, it has increased the training requirements and controls on appraisers. According to Hayes, under CMMI, the SEI reviews each appraisal that comes in for irregularities. And under CMMI, appraisers have to file a report called an Appraisal Disclosure Statement that clearly states which parts of the organization and projects were assessed, as well as all the people who took part in the assessment (though assessed companies are not required to reveal that report publicly, either). The SEI, along with the lead appraiser community, is also developing a "code of ethics" for appraisers.

Yet if CIOs want to get the true picture about appraisers, to check if they’ve ever been reprimanded for performing faulty assessments or thrown out altogether for cheating, they are out of luck. The SEI will not reveal any information about errant appraisers.

And the SEI has no intention of becoming a governing body like the American National Standards Institute (ANSI), which controls ISO 9000 certification in the United States. ANSI requires companies to be reassessed every six months if they want to maintain their ISO 9000 certification and reassesses all its appraisers each year. "No one has asked us to become a governing body, and that’s not our mandate. And if we did, what would that solve?" the SEI’s Humphrey asks. "It wouldn’t excuse anyone from doing their homework."

Indeed, CIOs who look to CMM for guarantees won’t find them, says Rick Harris, director of application development for OnStar, a division of GM that provides communications inside the company’s vehicles. He recalls confronting a manager from one of his CMM Level 5 offshore outsourcing companies who did not know how to do a testing plan for software. "My people had to train him to do it," he says. On another occasion, Harris’s staff discovered that the offshore provider had fallen far behind schedule in one of its projects but had not told him. "You’d think a Level 5 company would have told me months before that the schedule was slipping and we needed to do something," he says.

Problems like those can damage CIOs’ credibility inside IT and with the business?especially if they used a CMM level to defend a decision to move development offshore or use a particular outfit. As Harris has learned, what matters is what’s behind the impressive-looking number. Is there a verifiable commitment to quality, process and training? Can companies demonstrate improvements they’ve made over time in customer delivery times, developer productivity and defect density? Will the project managers that went through the assessment be assigned to your project? If the answer to any of these questions is no, then a CMM Level 5 isn’t worth much.

There is still no substitute for deep due diligence. "The real test is when you get into the trenches and see whether these companies bring their capabilities to bear," says Harris. "Do their people and processes hold up under pressure? In my experience, in some cases they have and others they haven’t."

Related:

Copyright © 2004 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
7 secrets of successful remote IT teams