Lessons from Hurricane Katrina: It Pays to Have a Disaster Recovery Plan in Place
"Monday was not a good day." That's how Entergy CIO Ray Johnson, not one for hyperbole, remembers Aug. 29, 2005, the day Hurricane Katrina roared ashore on the Gulf Coast. But Entergy's recovery efforts can be traced back to long before Katrina hit.
Around that time the decision was made to do a full implementation of the disaster recovery plan and Entergy’s core restoration team flew into action, led by Randy Helmick, vice president of customer service and support during normal operations but given the title of "storm boss" when an emergency is declared.
Johnson made it to Entergy’s storm command center a.k.a. "The Power House" in Jackson, Miss., around 4 a.m. on Sunday morning. Katrina had strengthened from category three to category five. "The news reports were alarming," Johnson says. "The potential implications for the city went up dramatically."
As outlined in the disaster recovery plan, Johnson’s team prepared the company’s systems that would be most critical in the restoration of electricity – its outage recording and management applications -- to run off the Little Rock data center in case something happened in Gretna.
Sure enough, by 3 a.m. Monday Gretna not only lost commercial power but the back-up generator was sustaining serious damage from wind and debris. It wasn’t a good sign. "We’ve had pretty bad storms before where we’ve lost commercial power and failed over to the generator," explains Johnson. "But we’ve never lost both." A day later he would find out the Gretna center suffered roof and water damage as well. As the morning began, Johnson declared an emergency with its vendor SunGard to reserve capacity at its hot site facility should they not be able to replicate systems in Little Rock.
Tuesday, the electricity was out everywhere -- even in Jackson. "Tuesday was a pretty rough day," Johnson says. "We didn’t even have power at the power house." That evening was the first chance for Johnson to send an expeditionary force to the Gretna data center where they discovered the extent of the damage.
A significant portion of Entergy’s IT staff actually works for SAIC as part of a major outsourcing relationship dating back to 1999. "When we sent people out to the data centers in those first few days, when there was no food or water yet, you couldn’t tell who wore what badge. It didn’t matter, we were all working together," Johnson says.
Many vendors went beyond the call of duty. "All of our vendors – and even some we’ve never worked with – were there within a couple of days saying, What do you need? How can we help?" recalls Johnson.
Although Entergy’s most critical applications were successfully brought online in Little Rock from backup tapes sent over the weekend, on Wednesday the team determined they could get the Gretna generator back online, bringing in a generator from another facility as backup. On Thursday, they brought in a contractor to patch up the roof and by Friday had it up and running again. Another storm was brewing – what would become Hurricane Ophelia – so Entergy continued on its path with SunGard in order to "keep all our options open," Johnson says. Ophelia took a different tack and by Labor Day, "we had all but completed our disaster recovery plan," recalls Johnson. All critical and medium-priority applications had been restored at the Gretna data center and the disaster recover team worked to continue to get all "normal" systems up and stabilized.
Hurricane Katrina


