As soon as Michael Baker Corp.\u2019s IT department finished installing a Web-based procurement system from ePlus, Bruce Higgins, CIO of the $405 million engineering and construction company, was bombarded with inquiries from colleagues about the system\u2019s effectiveness. He had ample anecdotal evidence suggesting that orders were getting turned around more quickly, but he didn\u2019t have tangible proof that the system was improving efficiency.So he decided to conduct his first ever post-implementation audit (PIA) of the new system. A PIA is a top-to-bottom evaluation of the hard and soft benefits derived from a strategic information system, the security of that system and the project management process for deploying it. From the PIA, Higgins learned that because IT miscalculated the number of people needing to use the new system, the ROI was driven down by the cost of ordering additional licenses. The PIA also showed that the system was saving the company more than $150,000 yearly, however, so Higgins was able to prove the system\u2019s worth to his colleagues. And he learned some valuable lessons on what not to do on subsequent projects.CIOs would do well to follow Higgins\u2019 lead in realizing that a PIA is a worthwhile way to prove the value of IT. But many don\u2019t. In fact, Barbara Gomolski, a research director with Gartner, estimates that a mere 20 percent of companies take the time to conduct PIAs.Companies avoid post-implementation audits for many reasons: They take too much time and drain away valuable personnel resources\u2014two things currently in short supply. They require reams of documentation so that processes and results can be validated. Finally, project sponsors and implementers fear that the results of an audit, if unfavorable, will be used against them.The CIOs at companies successfully performing audits have identified critical success factors, including: getting the right people involved, timing the audit properly, and collecting enough documentation to facilitate the smooth execution of a PIA. They pick the right projects to audit (for more information on this, see "How to Select the Right Project for Your First Post-Implementation Audit" at www.cio.com\/printlinks). And these CIOs share several key traits on how they ensure that PIAs become a sustained practice in their organizations: They are all committed to continuous improvement, they\u2019ve made PIAs a part of their project management methodologies, and they have their CEOs\u2019 support.But companies not performing PIAs are missing out on the important benefits such data provides. Although the challenges and risks associated with PIAs seem formidable, 66 percent of this year\u2019s CIO 100 honorees always or frequently conduct them, according to a CIO survey. Since the honorees were chosen for their exceptional resourcefulness, their prevalent use of the audit establishes it as a best practice among highly effective organizations. PIAs provide a thorough approach for proving the value of high-cost, mission-critical IT investments and for gleaning project management best practices, which CIOs can then apply to keep subsequent projects on track. At a time when the value of CIOs, IT departments and IT investments are under increased scrutiny, PIAs are now more critical to CIOs\u2019 success and survival than ever before. "With the pressure on business today and the responsibility of IT to help the business units understand where their dollars are being spent, I really have trouble with anyone saying you shouldn\u2019t be doing [PIAs] in this [hard economic] time," says Jim Smith, CIO of Sun Life Financial and a strong advocate of PIAs.How to Overcome Resistance to AuditsMany CIOs and their IT organizations shy away from audits because they think they\u2019ll be scapegoated if the audit reveals problems or lack of ROI. That\u2019s why Michael Barilla, vice president for business information services of Greif, a $1.6 billion manufacturer of industrial and paper packaging, frequently reminds himself and his IT staff that systems implementations and software deployments are not IT projects, but business projects. So if a project fails, he says, "it\u2019s going to fail as a team," and everyone\u2014not just IT\u2014will be accountable.Gartner\u2019s Gomolski concurs. If an audit exposes a shortcoming, she says, that problem shouldn\u2019t only reflect on IT, especially if the project was jointly sponsored by the business, IT and finance."Exposing problems can be a double-edged sword," says Gomolski. "But it\u2019s better to be proactive. If you haven\u2019t achieved the value you thought you would, figure out what you can do to change that."Another common concern about PIAs shared by CIOs, IT employees and even businesspeople is that they suck up too much time and require too much toil. But companies that perform PIAs say they shouldn\u2019t\u2014and don\u2019t\u2014take a lot of time to execute. Honeywell Aerospace aims to spend no more than seven to 10 days conducting an audit, and Sun Life Financial tries to complete PIAs within two weeks.Furthermore, CIOs who conduct PIAs say the time and resources audits use get recouped on subsequent project implementations. Resistance to audits also often stems from a desire to be rid of a project once the deployment is complete and to move on to the next challenge. The trick is to make it easy for workers to conduct and provide feedback for PIAs. For example, when Sun Life Financial\u2019s IT project office needs to solicit feedback on a system from business users, it asks them to fill out an electronic survey, on their own time, that takes no more than 20 minutes to complete. The response rate for those surveys is usually better than 50 percent and is sometimes close to 100 percent, says Ed Esposito, director of the project office.Because Honeywell Aerospace\u2019s IT organization for its Aviation Aftermarket Services unit also has trouble reengaging business users in a project during an audit, the IT organization leverages its staffers who have completed training in Six Sigma, a process improvement methodology. Those with Six Sigma training, who are experts in business processes, provide the IT department with the feedback it needs to determine whether a system has streamlined workflows.Engage the Right PeopleWho should perform PIAs is a matter of great debate. The most common groups of workers include one or more of the following.\n\n\n\n Members of the project implementation team from IT\n\n Members of the project implementation team from both IT and the business\n\n Representatives from a company\u2019s internal audit department\n\nAt Sun Life Financial, IT\u2019s project office leads the PIA process on its own IT and non-IT projects. But it does so in conjunction with the finance department and the company\u2019s internal audit department. IT projects are audited from the beginning and on an ongoing basis, rather than at the end of an implementation, which ensures that IT follows sound project methodologies, meets user requirements, stays on budget and implements proper security controls. \n\nSun Life Financial\u2019s approach comes closest to being the best, according to PIA experts. Don Christian, a partner with PricewaterhouseCoopers, says the PIA team should consist of a businessperson and an IT person who were involved with the implementation, and that it should be led by someone independent, such as an internal auditor, who was not part of the project team. Christian says it\u2019s better to have a group of people from different functions participate, rather than just an IT team or just internal audit because they all provide valuable input. The advantage of having members of the IT project team involved is that they\u2019re intimately familiar with the benefits, deliverables and requirements of the project. And because they know the project so well, it is easier for them to fully evaluate a project. Having a businessperson on the audit team is important because she can more easily determine if an external factor rather than a systems failure is causing a system to not generate expected value. And an independent auditor is important because he\u2019s not afraid to ask tough questions and will prevent the members of the project team who are involved in the audit from softening any findings.Time It RightWhen to conduct an audit, just as who should lead it, is a matter of debate. When to start depends on the type of system or application deployed, the amount of time it will take before the application begins generating some results or data, and the amount of time it takes staffers to get acclimated to the new system and new processes. Generally the audit should take place well within a year of implementation."When we started the audit a month after the implementation, we didn\u2019t have enough data to see if the system was successful or not," says Michael Baker Corp.\u2019s Higgins. "It\u2019s really important to make sure the system is up and running long enough to have enough data in the system that you can analyze."Ken Cunneen, IT leader for technology and integrated supply chain systems in Honeywell Aerospace\u2019s Aviation Aftermarket Services division, says if a company implements a financial system that only gives results once a quarter, the audit team should wait at least three to six months until the system has generated enough data before performing a PIA.On the other hand, the sooner you start the audit, the fresher the data and the easier it is to cull lessons learned. Which is why PIAs should not be a onetime event, says PWC\u2019s Christian.Sun Life Financial\u2019s project office begins PIAs within a month of project completion to do a postmortem on the implementation process and to get feedback from users to make enhancements to the system. "When we have benefits that are realized over a period of years after the project is implemented, we do follow-up reviews," says Sun Life Financial\u2019s Esposito.Higgins plans to take that iterative approach with Michael Baker Corp.\u2019s ERP implementation, which is scheduled to be completed by the end of this year. "We will begin to audit the effectiveness of the new system within a couple of months after we finish implementing it," says Higgins. "Then we\u2019ll probably start to look at the ROI six to eight months after we go live so that we have enough data to perform a good analysis."Maintain Meticulous DocumentationAnother key to successful PIAs is good documentation, which takes many forms. It includes the business case that outlines the system\u2019s expected cost, benefits and ROI; the project\u2019s time line, including key metrics and milestones; a breakdown of the system\u2019s technical requirements; a record of the security and financial controls that have been put inside the system; and a compendium of all the changes that have been made to the system or project plan. But getting good documentation is one of the biggest challenges CIOs and IT departments face in conducting PIAs, particularly if they\u2019re doing one for the first time.Sharon L. Thompson, director of IT audit for the AARP, says that if an auditor discovers that the implementation team does not have a thorough business case, a detailed project time line or meticulous records of changes and security controls, the auditor should note that discovery in the audit report and include a recommendation for the implementation team to keep better records.To make life easier for himself and his post-implementation auditors, Higgins uses a Web-based collaboration tool called MPOWR\u2014developed by members of his IT department to do version control and to archive changes made to the project\u2019s scope, templates, time line and budget. When it comes time to do a PIA, Higgins sends the auditors a link to the MPOWR site with a user name and password.Sun Life Financial\u2019s IT project office also developed a project portfolio management system in-house that the project manager uses to report on a monthly basis changes to a project\u2019s cost, milestones and schedule.Act on Lessons LearnedIf you don\u2019t use PIAs to identify ways to improve project management, you\u2019re not realizing their full value. Sun Life Financial\u2019s Esposito says project team members who served on the audit and implementation teams hold a meeting during the PIA process to discuss what went well and what went wrong both during the implementation and the audit, and to identify areas for improvement.To ensure that people take these lessons to heart, Sun Life Financial posts them on the corporate intranet and submits to senior management a summary of findings from PIAs of all projects done during the course of the year. Esposito also makes changes to project management and implementation processes if a project\u2019s PIA indicates that changes are warranted.Sun Life Financial annually reviews the audits of different projects to identify common areas that need improvement. Gartner\u2019s Gomolski says it\u2019s important to do this type of project-to-project comparison. "The real value," she says, "is in seeing some of the patterns. For example, \u2019We screw up every time we do an integration project. What does that mean?\u2019 Or, \u2019Every time we do a project with this business unit, we suffer from amazing scope creep because they don\u2019t really know what they want, so maybe we have to work with them differently.\u2019"Sustain the MomentumNow that you\u2019re armed with convincing arguments on the importance of PIAs and success factors for pulling them off, making them a sustained practice in your organization should look less daunting."If you have the project\u2019s objectives, approach, how it\u2019s going to be organized, how the business areas are involved in the project, the expected costs, and the anticipated hard and soft benefits well-defined up front, and you manage change along the way, then when you come in to do the post-project audit, there are no surprises," says Esposito.Not only do PIAs go smoother when you\u2019ve done all the hard work on the front end, but you increase your odds of actualizing the system\u2019s value. "By getting a better understanding of project costs up front," says Higgins, "you\u2019ll be more likely to hit the ROI you\u2019re trying to meet."