CIO — 1. Who was the appraiser? How many Level 5 assessments has he done? (If, for example, the appraiser had never evaluated a Level 5 company before, he will not be well qualified to know what a Level 5 company really looks like.)
2. What part of the company was tested? "Enterprise CMM" companies may have had 10 percent or less of their projects assessed to get the level.
3. How long ago was this done? If it’s more than two years old, chances are it has little meaning because the company will have changed and, in the case of offshore companies, may have grown many times the size it was when assessed.
4. How long did it take the company to move from one level to another? If it’s fewer than two years between Levels 1 through 4, or fewer than one year between Levels 4 and 5, that does not fit with the average. Check the company out further.
5. Where is your evidence of continuous improvement? CMM Level 5 means continuous improvement?make the company show you its improvement history.
6. Who runs the quality group? Ask to meet the quality organization that monitors and audits the process, and make sure it isn’t ghettoized. There should be more than a handful of people, and they should have the power in the organization to change things. They should report directly to the CEO, and the company should link the quality system to executive accountability and compensation.
7. Was the appraiser from inside or outside the organization? Internal assessors cannot be as objective as external lead assessors who have no links to the company.
8. Where are the reports? Ask for formal documents that the assessor must provide to the company, most importantly the Final Findings Report (discusses strengths and weaknesses of the company) and, for CMMI, the Appraisal Disclosure Statement (states exactly where, when, how and by whom the assessment was done). If the documents are vague or fluffy, ask for more detail. The company has it. If the company won’t let you see the reports, move on. You may have found a cheater.
9. What types of projects were assessed? If you’re a financial services organization, you want to make sure that at least one of the projects assessed had something to do with financial processes.
10. Did the appraiser consult on the processes being assessed? Appraisers are supposed to be independent and objective. If they’ve helped the company develop the processes they are rating, it’s a conflict of interest, and they may be tempted to improve the ratings to get their consulting fees.
11. How many project managers who were assessed at Level 5 will be on your project team?
12. How does the company train new people to be CMM Level 5? If the company doesn’t train well, it will not sustain its CMM level for long.
