Applied Insight – Tracks in the Snow

There’s a tradition in IT project management that says if a project comes in on time, on budget and with the required features and functions, it is considered successful. Meeting that standard is difficult enough. A continuing study of thousands of IT projects by The Standish Group consultancy found that in 2004, just 29 percent of projects were successful.

But this definition of success needs revising. In performing 72 IT project retrospectives at 57 organizations since 1999, graduate students in the Master of Science in the Management of Information Technology program at the University of Virginia’s McIntire School of Commerce have discovered that projects that were found to meet all of the traditional criteria for success—time, budget and specifications—may still be failures in the end because they fail to appeal to the intended users or because they ultimately fail to add much value to the business.

When Success Is Failure

We call these “failed successes.” For example, a real estate management company successfully completed a two-year project to develop a Lotus Notes–based CRM application that was designed to provide strategic advantage in leasing vacant space. The project met all the project specifications but didn’t successfully integrate with the company’s business processes. So in the end, no one used it. The company’s CTO put it best: “The application is 100 percent effective at 100 percent participation and zero percent effective at 95 percent participation.”

Similarly, projects considered failures according to traditional IT metrics may wind up being successes because despite cost, time or specification problems, the system is loved by its target audience or provides unexpected value. For example, at a financial services company, a new system to enable rapid development, testing, deployment and measurement of collections strategies and to improve collections performance was six months late and cost more than twice the original estimate (final cost was $5.7 million). But the project ultimately created a more adaptive organization (after 13 months) and was judged to be a great success—the company had a $33 million reduction in write-off accounts, and the reduced time-to-value and increased capacity resulted in a 50 percent increase in the number of concurrent collection strategy tests in production.

New Criteria for Success

Indeed, the true impact of a project may not be felt until long after installation is complete. The study data showed that beyond the traditional metrics of time, cost and product (an acceptable system of reasonable quality that meets specifications), the following additional criteria need to be applied to projects to determine long-term success.

Use: The project’s resultant products or services were used by its intended constituents.

Value: The project led directly to the organization’s improved efficiency or effectiveness (common metrics included NPV, IRR, EVA and the Balanced Scorecard).

Learning: The project increased stakeholder knowledge and better prepared the organization for future challenges.

Interestingly, the retrospective data showed that across all stakeholders (project managers, their teams, project sponsors, top management and users), the top three criteria by which stakeholders judged a project’s success were product, use and value, respectively. Meanwhile, cost was ranked lowest overall. None of the groups ranked learning (preparing for the future) in their top three criteria, although all of them suggested that learning was of at least moderate importance.

Although project managers and team members seemed to be more concerned with the traditional measures—in other words, getting the project done according to specification, on time and on budget—users, not surprisingly, cared most about whether the system would be used as intended. It’s hard to get this perspective during the course of the project. The issues that matter most to users—use and value—can’t be resolved until after the project is complete in the traditional sense. If maximizing overall stakeholder satisfaction is important, top managers should be focusing more on the product, use and value criteria.

This is why it’s critical for CIOs to review projects both during and after installation if they are to build and maintain a long-term project success rate. Based on our six years of experience, retrospectives should be conducted after completing each major milestone in large projects, and one to two weeks after implementation. Benefits realization should be conducted later, when business value metrics can legitimately be assessed.

The Benefits of Looking Back

If done right, retrospectives offer a variety of potential benefits. Effective reviews take stakeholders beyond their own personal impressions of the project and present a larger view of the organizational impact—a project that may have seemed “a nightmare” to an overworked IT project manager may actually be tremendously valuable to the organization. Reviews also create opportunities to make improvements in processes, procedures and culture—rather than treating each project as a one-off, in which the lessons of earlier projects are ignored or forgotten. Better estimating and scheduling of future projects are possible because the reviews capture historical data on the true size, effort and time involved in past projects.

Finally, project reviews have an invaluable human aspect. If a project went badly, people can acknowledge and repair relationship issues so that future projects can go better. If the project was a success, the review lets you reflect on accomplishments before going on to the next problem.

Project reviews expand the concept of the project time line. It can be difficult to adapt to the new way of thinking. Project managers must negotiate the trade-offs between process (time, cost and product) and outcome (use, learning and business value). The project charter should include negotiated success metrics, the project dashboard should enable real-time metrics, and the project retrospective should document the actual results, concluding with overall stakeholder satisfaction.

IT struggles with a continuing record of project failure. Yet the key to improving the rate of project success may lie right underneath managers’ noses, in the form of comprehensive project retrospectives. There is a lot to be gained from looking into the rearview mirror from time to time.