Software Quality: Bursting the CMM Hype

CIO — As soon as she walked into the meeting, Jane Smith knew that the executive on the other side of the desk wanted to buy something that Smith wasn’t supposed to sell: a trumped up rating for the executive’s software development division so that his company could qualify to bid on contracts from the United States Department of Defense.

Smith (not her real name) is one of a select group of experienced IT pros, called lead appraisers, who go into companies and assess the effectiveness of their software

development processes on a scale from 1 (utter chaos) to 5 (continuously improving) under a system known as the Capability Maturity Model, or CMM. The company she was visiting wanted to move up to Level 2, but based on some initial discussions, Smith knew that the company was a 1. Level 1 describes most of the software development organizations in the world: no standard methods for writing software, and little ability to predict costs or delivery times. Project management consists mostly of ordering more pizza after midnight.

After a few initial niceties, the executive leaned across the table to Smith and another lead appraiser who had accompanied her to the meeting and asked, "How much for a Level 2?"

"That’s when I got up and left the room," Smith recalls. "The other appraiser stayed. And the company got its rating."

The stakes for a good CMM assessment have gotten only higher since Smith’s close encounter with corruption some 10 years ago. Today, many U.S. government agencies in addition to the DoD insist that companies that bid for their business obtain at least a CMM Level 3 assessment?meaning the development organization has a codified, repeatable process for an entire division or company. CIOs increasingly use CMM assessments to whittle down the lists of dozens of unfamiliar offshore service providers?especially in India?wanting their business. For CIOs, the magic number is 5, and software development and services companies that don’t have it risk losing billions of dollars worth of business from American and European corporations.

"Level 5 was once a differentiator, but now it is a condition of getting into the game," says Dennis Callahan, senior vice president and CIO of Guardian Life Insurance. "Having said that, there are some Level 3 or 4 startups that we might consider, but they have a lot more convincing to do before I would do business with them. They would be at a disadvantage."

With CIOs increasingly dependent on outside service providers to help with software projects, some have come to view CMM (and its new, more comprehensive successor, CMM Integration, or CMMI) as the USDA seal of approval for software providers. Yet CIOs who buy the services of a provider claiming that seal without doing their own due diligence could be making a multimillion-dollar, career-threatening mistake.

Continue Reading