Comair's Christmas Disaster: Bound To Fail

The 2004 crash of a critical legacy system at Comair is a classic risk management mistake that cost the airline $20 million and badly damaged its reputation.

1 2 Page 2
Page 2 of 2

And another problem was looming. As it turned out, the crew management application, unbeknownst to anyone at Comair, could process only a set number of changes—32,000 per month—before shutting down. And that’s exactly what happened. On Christmas Eve, all the rescheduling necessitated by the bad weather forced the system to crash. As a result, Comair had to cancel all 1,100 of its flights on Christmas Day, stranding tens of thousands of passengers heading home for the holidays. It had to cancel nearly 90 percent of its flights on Dec. 26, stranding more.

There was no backup system. It took a full day for the vendor to fix the software. But Comair was not able to operate a full schedule until Dec. 29.

The big lesson: Manage your risk

The Comair disaster is a classic case study in operational risk, according to industry experts, who say both Comair and Delta carry some blame. Comair IT executives should have done the kind of risk management analysis that would have alerted Delta to the dangers of not replacing the legacy system sooner. And IT should have repeatedly brought that analysis to the attention of Delta officials until a replacement system was funded. Similarly, Delta executives should have insisted on scrutinizing Comair’s operations and done their own analysis of the carrier’s risks.

"Anything that can damage a parent company’s brand or reputation has to be managed in some way," the former Delta IT executive says. "Risk assessment of worst-case scenarios at Comair should have happened at Delta."

What happened at Comair is hardly an isolated problem. Old green-screen legacy systems exist at the core of many businesses, and they can’t take the velocity and number of transactions coming at them today from outside. "The more applications a legacy system is hardwired to over the years, the more fragile it becomes," says EDS’s Charlie Feld, also a former Delta CIO.

The larger problem is that operational risks are not introduced into day-to-day decision making at many companies, such as Comair, the way things such as the mechanics of planes and daily operations are. Robert Charette, director of the Cutter Consortium’s enterprise risk management and governance practice, says Comair executives still don’t seem to have learned this basic risk management lesson. As late as March, Comair’s Miller was blaming the debacle on bad weather. "If the weather had not hit as hard as it did, the problem would have never come up," he says.

Industry observers acknowledge that convincing first Comair corporate and then Delta executives of the necessity of replacing the system would have been a tough sell for IT. But if Comair’s IT had done a cost-benefit analysis of the risk of not replacing the system, they could have made a convincing business case for an upgrade, says a former Delta IT executive.

The technology is there, Feld says. "The question is how people in IT lay out that multiyear plan and get the right partners in there to help transform these legacy systems," he adds. "Because if you don’t, there will be more meltdowns."

The epilogue

Having lost nearly as much as Comair made in profit the previous quarter with this fiasco, Delta finally intervened. On Jan. 17 of this year, 20-year Comair veteran and President Randy Rademacher stepped down, and Delta assigned Fred Buttrell, CEO of Delta Connection (which manages Delta’s network of regional carriers) to take over. Shortly after Rademacher’s resignation, Stuart was also asked to leave. Some say more Delta blood may be infused into this regional subsidiary.

But whether Delta will invest more in Comair’s IT remains to be seen. In its 2004 annual report, Delta said that it will post another substantial loss in 2005. A bankruptcy filing remains a possibility. And, says Childress, "when the airlines are in trouble, it’s a lot harder to find cash for IT renewal and replacement." In fact, Delta has not ruled out the possibility of selling Comair or the other regional airline it owns to raise cash.

In the meantime, Bardes says he meets up with some of his old Comair coworkers for lunch every now and then. "I’ll say, Are you still keeping your head down? They’ll say, Oh yeah," Bardes says. "That place just seems to punish people who want to be agents of change."

As of March, Comair was still using the nearly 20-year-old crew management system from SBS, though with a lot more care. SBS implemented a bridge solution, dividing the legacy system into two modules—one for pilot schedule and another for flight attendant schedules—each with a 32,000 monthly limit of its own. Comair also began generating a daily report to monitor the volume of transactions going through the system.

And plans are still in the works to replace it.

Copyright © 2005 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
Download CIO's Roadmap Report: 5G in the Enterprise