Lessons from Hurricane Katrina: It Pays to Have a Disaster Recovery Plan in Place
"Monday was not a good day." That's how Entergy CIO Ray Johnson, not one for hyperbole, remembers Aug. 29, 2005, the day Hurricane Katrina roared ashore on the Gulf Coast. But Entergy's recovery efforts can be traced back to long before Katrina hit.
CIO — "Monday was not a good day."
That's how Ray Johnson, not one for hyperbole, remembers Aug. 29, 2005 -- the day Hurricane Katrina roared ashore on the Gulf Coast. But there’s a reason for the understatement. Johnson, CIO of New Orleans-based Entergy, would see darker days to follow.
Johnson has had little time over the last two and a half weeks to reflect on his company's efforts not only to restore power to its customers affected by the hurricane but also to prevent severe business interruption despite a virtual attack on its own physical and human infrastructure. But he spoke to CIO on Sept. 15 to discuss Entergy's preparation for and response to the worst disaster in the company's history.
Entergy’s recovery efforts can be traced back to long before Katrina hit. The $10 billion energy company has a disaster recovery plan that's tested and revised once a year. Last year, that "test" came in the form of Hurricane Ivan. Entergy activated its storm command center and disaster recovery processes in response to the approaching storm which ultimately made landfall further east in Pensacola, Fla. And just this April, the company conducted an extensive storm drill which simulated what corporate executives thought of as a worse case scenario -- a major hurricane followed by extreme flooding.
"This is not a unique event for us," explains Johnson, "I wish it was. But we've got our disaster plan nailed. We review it at least once a year and either we conduct a drill or get to test it when a hurricane threatens us and then misses."
On Friday, August 26, it looked like Johnson would have another opportunity to test the disaster recovery plan. Katrina was crossing over Florida and beginning to churn in the Gulf. But early in the day, the hurricane was predicted to hit several hundred miles east of New Orleans.
As Friday evening wore on, the situation changed. This was not a test. "We started making our storm calls and made the decision to activate our disaster recovery plan," which calls for some preliminary actions to be taken beginning 72 hours before a category 3 hurricane is scheduled to make landfall," recalls Johnson.
Saturday morning 5 a.m., Johnson sent his first "away" team to the active disaster recovery site Entergy has in Little Rock, Ark. Entergy's primary data center is located in Gretna, La., across the river from Entergy's corporate headquarters. The back-up generators supporting that data center had never failed before but Johnson knew there could be one of many gloomy firsts he might encounter this time out. "By late in the day (on Saturday) it became more and more obvious as Katrina tracked closer and closer that New Orleans was going to be a primary target." Once the first team arrived safely, a second was deployed.
