When Eric Bardes joined the Comair IT department in 1997, one of the very first meetings he attended was called to address the replacement of an aging legacy system the regional airline utilized to manage flight crews. The application, from SBS International, was one of the oldest in the company (11 years old at the time), was written in Fortran (which no one at Comair was fluent in) and was the only system left that ran on the airline\u2019s old IBM AIX platform (all other applications ran on HP Unix). \n\nSBS came in to make a pitch for its new Maestro crew management software. One of the flight crew supervisors at the meeting had used Maestro, a first-generation Windows application, at a previous job. He found it clumsy, to put it kindly. "He said he wouldn\u2019t wish the application on his worst enemy," Bardes recalls. The existing crew management system wasn\u2019t exactly elegant, but all the business users had grown adept at operating it, and a great number of Comair\u2019s existing business processes had sprung from it. The consensus at the meeting was that if Comair was going to shoulder the expense of replacing the old crew management system, it should wait for a more satisfactory substitute to come along. \n\nAnd wait they did. The prospect of replacing the ever-maturing crew management system was floated again the following year, with plans laid out to select a vendor in 2000. But that didn\u2019t happen. Over the next several years, Comair\u2019s corporate leadership was distracted by a sequence of tumultuous events: managing the approach of Y2K, the purchase of the independent carrier by Delta in 2000, a pilot strike that grounded the airline in 2001, and finally, 9\/11 and the ensuing downturn that ravaged the airline industry. \n\n\nA replacement system from Sabre Airline Solutions was finally approved last year, but the switch didn\u2019t happen soon enough. Over the holidays, the legacy system failed, bringing down the entire airline, canceling or delaying 3,900 flights, and stranding nearly 200,000 passengers. The network crash cost Comair and its parent company, Delta Air Lines, $20 million, damaged the airline\u2019s reputation and prompted an investigation by the Department of Transportation. \n\nChances are, the whole mess could have been avoided if Comair or Delta had done a comprehensive analysis of the risk that this critical system posed to the airline\u2019s daily operations and had taken steps to mitigate that risk. But a look inside Comair reveals that senior executives there did not consider a replacement system an urgent priority, and IT did little to disrupt that sense of complacency. Though everyone seemed to know that there was a need to deal with the aging applications and architecture that supported the growing regional carrier\u2014and the company even created a five-year strategic plan for just that purpose\u2014a lack of urgency prevailed. \n\n\nAfter the acquisition by Delta, former employees say Comair IT executives didn\u2019t do the kind of thorough management analysis that might have persuaded the parent airline to invest in a replacement system before it was too late. Instead, Delta kept a lid on capital expenditures at Comair, with unfortunate consequences. The failure of the almost 20-year-old scheduling system not only saddled Delta with a plethora of customer service and financial headaches that the airline could ill afford but it also provides a cautionary tale for any company that thinks it can operate on its legacy systems for just...one...more...day. \n\n\n\nThe five-year plan that wasn\u2019t \n\nToday, Cincinnati-based Comair is a regional airline that operates in 117 cities and carries about 30,000 passengers on 1,130 flights a day, with three or four crew members on each. But back in 1984, when Jim Dublikar joined the company as director of finance and risk management, Comair had just 25 airplanes and not a jet among them. Dublikar, who served as Comair\u2019s director of risk management and information technology from 1992 until 1999, explains that back then, flight crew managers did all their scheduling with pen and paper. But in 1986, Comair leased software from SBS that kept track of crews, the flights they were assigned to and how many hours they were flying, in order to be in compliance with union and federal regulations. \n\n\nThe system worked just fine. \n\n\nIn 1993, Comair bought a jet\u2014the first Bombardier CRJ regional jet in the industry, in fact. The company grew swiftly. But by 1996, other regional contenders such as American Eagle, Mesa and Continental Express acquired their own jets, and Comair lost its competitive advantage. "At that point, the playing field had gotten pretty even," says Dublikar, now an airline consultant. "So we had to start looking at ways of doing things better and more efficiently." \n\nOver the years, Comair, like most airlines, had acquired a hodgepodge of applications\u2014from crew scheduling to aircraft maintenance to passenger booking engines. "We had several systems that were getting pretty long in the tooth\u2014around for seven, eight, nine years," Dublikar says. \n\nUnfortunately, you can\u2019t see a crew management system age the way you can see an airplane rust. But they do. "These systems are just like physical assets," says Mike Childress, former Delta CTO and now vice president of applications and industry frameworks for EDS. "They become brittle with age, and you have to take great care in maintaining them." \n\n\nIn 1998, Dublikar and his IT steering committee brought in consultants from Sabre, the Southlake, Texas-based airline software and consulting company, to create a long-term IT strategy to address the issue of legacy systems and architecture. The consultants spent five months meeting with IT\u2019s various constituents in the business to find out what their needs were. They examined the airline\u2019s existing IT infrastructure and suggested a five-year strategic plan outlining (among other things) which systems needed to be retired, replaced or added, and a time line for doing so. \n\n\nThe crew scheduling system was marked for retirement. SBS was no longer "the only game in town," recalls Dublikar, and the case for replacing the system was pretty easy to make. "The application was getting old. There was risk there, and there was new technology out there," he adds. "There were even financial benefits to replacing the system in terms of crew productivity and expenses that could be controlled better in a new system." \n\n\nBut this was 1998, and for the next two years, Y2K absorbed most of IT\u2019s attention. By 1999, a significant amount of the work that had been laid out in the five-year plan (including Y2K remediation) had been completed or was under way\u2014including implementing an e-ticketing system, upgrading the corporate network, replacing the maintenance and engineering system (another high-risk legacy system written in Cobol), and implementing a revenue management application. \n\nThe replacement of the crew scheduling system was among those next on the list. But after nearly 15 years in use, the business had grown accustomed to the SBS system, and much of Comair\u2019s crew management business processes had grown directly out of it. Just look at a pilot\u2019s contract at Comair; the definition of a workday is lifted straight out of the old SBS crew management application and expressed in Julian minutes the way the system did. (There are 44,640 Julian minutes in a 31-day month.) "That\u2019s the reason why it\u2019s almost impossible to replace these systems," says John Parker, former airline CIO and 17-year Delta veteran, now CIO of A.G. Edwards & Sons. \n\n\nBut systems requirements had been defined, and the IT department was in the software selection process. Final vendor selection was slated for 2000, according to Dublikar. \n\nBut then, in the middle of 1999, Dublikar left Comair, and shortly afterward, Delta announced its plans to acquire Comair. \n\n\nHands off at Delta\n\nTo Delta, buying Comair\u2014one of its most profitable regional partners\u2014was a no-brainer. It made money. It was an industry leader in on-time, cancellation and missed baggage statistics. And it was a stock market darling. "Before Delta came in, Comair was one of the best managed and most successful regionals in the country," says Holly Hegeman, an airline industry analyst and founder of PlaneBusiness.com. "They had the reputation of being the top dog." \n\nBy all accounts, Delta\u2019s attitude toward Comair was, Why mess with success? Even IT looked okay to Delta. On paper, Comair had project time lines, good budgets, everything you\u2019d expect to see. "So there was no mandate at the Delta level to get the Comair IT ship righted," Hegeman says. The only area Delta appeared to be concerned about was marketing; the parent airline replaced the entire marketing department at Comair within days of taking over. \n\nComair, like most acquired companies, wasn\u2019t exactly welcoming to its new owner either. "There was definite friction," Hegeman says. "Top management at Comair didn\u2019t take kindly to being part of mother Delta." So Comair continued to run independently for the most part\u2014although as a wholly owned subsidiary, all major capital expenditures had to be approved by the parent company, Bardes and others say. \n\nAfter Dublikar left, the IT director position stayed vacant for a number of months. In early 2000, Mike Stuart, senior vice president of flight operations, was given oversight of IT. And in March 2000, Sherri Kurlas-Schalk, who had been with the company since 1990, was named IT director. The tendency in the IT department, meanwhile, was to "keep your head down" and not draw too much attention to anything, according to Bardes, who left Comair in late 2003 to join software and IT services company Compuware as a senior systems designer. After the uncertainty in IT leadership and the takeover by Delta, there was a palpable lack of commitment to projects in IT. "Everyone was expecting someone else to move projects along," Bardes says. "The business units were expecting IT to push a project through. And IT was waiting for the business unit to push it through." \n\nThe five-year plan, which was supposed to be revisited on a regular basis, languished. \n\nIn 2001, an 89-day pilot\u2019s strike from March to June shut down Comair and the Cincinnati\/Northern Kentucky International Airport, where Delta and Comair operate 90 percent of all flights. Comair closed its Cincinnati concourse, losing more than 800 daily flights and saddling Delta with a $200 million loss for the quarter. Once the strike was over, Comair\u2019s flight operations group, the primary users of the crew scheduling system, had their hands full getting planes back in the air. "You can\u2019t just switch things on and get an airline running again," says Bardes. "There\u2019s a lot of inertia and momentum lost when you shut down, and it\u2019s hard to get started again." During this period, they gave little or no thought to replacing the crew scheduling system. \n\nThen came 9\/11, crippling airlines big and small, and pushing some of the largest carriers into bankruptcy. Though Delta has thus far avoided that fate, the airline lost nearly $8.5 billion over the following four years, increasing the pressure to keep costs down. While there\u2019s no evidence that Delta refused to fund an upgrade for the crew scheduling system, "approval for capital expenditures seldom went through the first time [at Delta]," Bardes says. "They\u2019d want more analysis. They had a definite influence on how fast money got spent." \n\nDelta declined to comment for this story and referred questions to Comair. Comair spokesman Nick Miller would say only, "We\u2019ve been very straightforward in acknowledging the challenging time we\u2019ve been facing in the airline industry and that we\u2019ve had to be very prudent in how we\u2019ve invested in technology." \n\n\nLate in 2002, the Comair IT group did turn its attention back to the crew management system and brought in several vendors, including Sabre and SBS, to perform demos. Comair went down the road a bit with one vendor, which it refuses to name, but ultimately backed out during contract negotiations due to pricing concerns. All in all, there seemed to be no hurry on either Comair or Delta\u2019s part to get the project rolling, even though the crew scheduling system was (and still is) the oldest application of its kind still running at a regional carrier, according to a recent survey by Regional Aviation News. \n\n\nFinally, Comair got approval from Delta to replace the legacy SBS system and inked a deal with Sabre in June 2004 to implement its AirCrews Operations Manager. Implementation was set to begin in 2005. But by then, it would be too late. \n\n\nOn Dec. 16, Comair reported an operating profit of $25.7 million in the third quarter of 2004. A week later, a severe winter storm hit the Ohio Valley. The snow came with sleet and freezing rain. Deicing the jets took much longer than expected and some jets\u2019 tires froze to the ground. From Dec. 22 through the 24th, Comair had to cancel or delay 91 percent of its flights. \n\nAnd another problem was looming. As it turned out, the crew management application, unbeknownst to anyone at Comair, could process only a set number of changes\u201432,000 per month\u2014before shutting down. And that\u2019s exactly what happened. On Christmas Eve, all the rescheduling necessitated by the bad weather forced the system to crash. As a result, Comair had to cancel all 1,100 of its flights on Christmas Day, stranding tens of thousands of passengers heading home for the holidays. It had to cancel nearly 90 percent of its flights on Dec. 26, stranding more. \n\n\nThere was no backup system. It took a full day for the vendor to fix the software. But Comair was not able to operate a full schedule until Dec. 29. \n\n\n\nThe big lesson: Manage your risk \n\nThe Comair disaster is a classic case study in operational risk, according to industry experts, who say both Comair and Delta carry some blame. Comair IT executives should have done the kind of risk management analysis that would have alerted Delta to the dangers of not replacing the legacy system sooner. And IT should have repeatedly brought that analysis to the attention of Delta officials until a replacement system was funded. Similarly, Delta executives should have insisted on scrutinizing Comair\u2019s operations and done their own analysis of the carrier\u2019s risks. \n\n\n"Anything that can damage a parent company\u2019s brand or reputation has to be managed in some way," the former Delta IT executive says. "Risk assessment of worst-case scenarios at Comair should have happened at Delta." \n\n\nWhat happened at Comair is hardly an isolated problem. Old green-screen legacy systems exist at the core of many businesses, and they can\u2019t take the velocity and number of transactions coming at them today from outside. "The more applications a legacy system is hardwired to over the years, the more fragile it becomes," says EDS\u2019s Charlie Feld, also a former Delta CIO. \n\n\nThe larger problem is that operational risks are not introduced into day-to-day decision making at many companies, such as Comair, the way things such as the mechanics of planes and daily operations are. Robert Charette, director of the Cutter Consortium\u2019s enterprise risk management and governance practice, says Comair executives still don\u2019t seem to have learned this basic risk management lesson. As late as March, Comair\u2019s Miller was blaming the debacle on bad weather. "If the weather had not hit as hard as it did, the problem would have never come up," he says.\n\nIndustry observers acknowledge that convincing first Comair corporate and then Delta executives of the necessity of replacing the system would have been a tough sell for IT. But if Comair\u2019s IT had done a cost-benefit analysis of the risk of not replacing the system, they could have made a convincing business case for an upgrade, says a former Delta IT executive. \n\n\nThe technology is there, Feld says. "The question is how people in IT lay out that multiyear plan and get the right partners in there to help transform these legacy systems," he adds. "Because if you don\u2019t, there will be more meltdowns." \n\n\nThe epilogue \n\nHaving lost nearly as much as Comair made in profit the previous quarter with this fiasco, Delta finally intervened. On Jan. 17 of this year, 20-year Comair veteran and President Randy Rademacher stepped down, and Delta assigned Fred Buttrell, CEO of Delta Connection (which manages Delta\u2019s network of regional carriers) to take over. Shortly after Rademacher\u2019s resignation, Stuart was also asked to leave. Some say more Delta blood may be infused into this regional subsidiary. \n\nBut whether Delta will invest more in Comair\u2019s IT remains to be seen. In its 2004 annual report, Delta said that it will post another substantial loss in 2005. A bankruptcy filing remains a possibility. And, says Childress, "when the airlines are in trouble, it\u2019s a lot harder to find cash for IT renewal and replacement." In fact, Delta has not ruled out the possibility of selling Comair or the other regional airline it owns to raise cash. \n\nIn the meantime, Bardes says he meets up with some of his old Comair coworkers for lunch every now and then. "I\u2019ll say, Are you still keeping your head down? They\u2019ll say, Oh yeah," Bardes says. "That place just seems to punish people who want to be agents of change." \n\n\nAs of March, Comair was still using the nearly 20-year-old crew management system from SBS, though with a lot more care. SBS implemented a bridge solution, dividing the legacy system into two modules\u2014one for pilot schedule and another for flight attendant schedules\u2014each with a 32,000 monthly limit of its own. Comair also began generating a daily report to monitor the volume of transactions going through the system. \n\nAnd plans are still in the works to replace it.