Lessons from Hurricane Katrina: It Pays to Have a Disaster Recovery Plan in Place

"Monday was not a good day." That's how Entergy CIO Ray Johnson, not one for hyperbole, remembers Aug. 29, 2005, the day Hurricane Katrina roared ashore on the Gulf Coast. But Entergy's recovery efforts can be traced back to long before Katrina hit.

PAGE 2

Around that time the decision was made to do a full implementation of the disaster recovery plan and Entergy’s core restoration team flew into action, led by Randy Helmick, vice president of customer service and support during normal operations but given the title of "storm boss" when an emergency is declared.

Johnson made it to Entergy’s storm command center a.k.a. "The Power House" in Jackson, Miss., around 4 a.m. on Sunday morning. Katrina had strengthened from category three to category five. "The news reports were alarming," Johnson says. "The potential implications for the city went up dramatically."

As outlined in the disaster recovery plan, Johnson’s team prepared the company’s systems that would be most critical in the restoration of electricity – its outage recording and management applications -- to run off the Little Rock data center in case something happened in Gretna.

Sure enough, by 3 a.m. Monday Gretna not only lost commercial power but the back-up generator was sustaining serious damage from wind and debris. It wasn’t a good sign. "We’ve had pretty bad storms before where we’ve lost commercial power and failed over to the generator," explains Johnson. "But we’ve never lost both." A day later he would find out the Gretna center suffered roof and water damage as well. As the morning began, Johnson declared an emergency with its vendor SunGard to reserve capacity at its hot site facility should they not be able to replicate systems in Little Rock.

Tuesday, the electricity was out everywhere -- even in Jackson. "Tuesday was a pretty rough day," Johnson says. "We didn’t even have power at the power house." That evening was the first chance for Johnson to send an expeditionary force to the Gretna data center where they discovered the extent of the damage.

A significant portion of Entergy’s IT staff actually works for SAIC as part of a major outsourcing relationship dating back to 1999. "When we sent people out to the data centers in those first few days, when there was no food or water yet, you couldn’t tell who wore what badge. It didn’t matter, we were all working together," Johnson says.

Many vendors went beyond the call of duty. "All of our vendors – and even some we’ve never worked with – were there within a couple of days saying, What do you need? How can we help?" recalls Johnson.

Although Entergy’s most critical applications were successfully brought online in Little Rock from backup tapes sent over the weekend, on Wednesday the team determined they could get the Gretna generator back online, bringing in a generator from another facility as backup. On Thursday, they brought in a contractor to patch up the roof and by Friday had it up and running again. Another storm was brewing – what would become Hurricane Ophelia – so Entergy continued on its path with SunGard in order to "keep all our options open," Johnson says. Ophelia took a different tack and by Labor Day, "we had all but completed our disaster recovery plan," recalls Johnson. All critical and medium-priority applications had been restored at the Gretna data center and the disaster recover team worked to continue to get all "normal" systems up and stabilized.

Hurricane Katrina

Loading...
 
SPONSORED LINKS
 

Making Consumer Two-Factor Authentication Simple and Cost-Effective

Mining the Cloud to Ease the Enterprise Compliance Burden

Solve Five Key IT Security Challenges with Cloud-Based Authentication

White Paper: Managed Security for a Not-So-Secure World

Secure Email and Web-Based Communication from Evolving Attacks

WagerWorks Takes Fraudsters Out of the Game using iovation

White Paper: A Security Blueprint Delivered From within the Network

Return on Information: Google Enterprise Search pays you back

ROI of Application Delivery Controllers

Webcast: Unleashing the Power of Customer Data

Disciplined Autonomy: Resolving the Tension Between Flexibility and Control

Enterprise Capture: Your Onramp to Business Process Automation

Cloud Computing--What is its Potential Value for Your Company?

Seven Design Requirements for Web 2.0 Threat Protection

Adobe® LiveCycle® solutions for business process automation

10 Ways Excel Drives More Value from Your SAP Investment

The Key to Proving and Improving the Value of IT to the Company

Unleash the Power of Java with Oracle JRockit Real Time

Taking the Service Desk to the Next Level

Return on Information: Google Enterprise Search pays you back. Get the facts.

VMware. The source for Business Infrastructure Virtualization.

ShoreTel tells businesses to untangle from competitors' complexity and turn to its brilliantly simple UC solution

See how AT&T can help protect your network.

Streamline IT Costs. Boost Performance with WAN Optimization.

Build your 1st app FREE with Force.com

Authentication as a Service by Forrester Research

Cloud-Based Authentication for Next-Generation Extranets

Mobile Security: The Essential Ingredient for Today's Enterprise

IDC White Paper: CCM for IT Compliance and Risk Management

Keeping Your Members Safe from Online Scams and Predators

Learn about the growing threat of insider data theft.

Upgrading to VMware vSphere with vWire

Maximizing website Return on Information with high-quality search

Gartner Magic Quadrant, Application Delivery Controllers 2009

Learn How Web Site Performance Impacts Shopper Behavior

Build a Foundation for Unified Communications

Removing the Barriers to IT Governance: How On-Demand Software Changes the Game

Should Your Email Live In The Cloud? A Comparative Cost Analysis

How Consumerization of IT Will Make Your Business More Productive

How does a software company save big with Green IT?

Translate business strategy into IT strategy and obtain maximum benefits.

eBook: How Can You Make Your People Productive Anywhere?

Mind the Talent Gap: Global Survey on IT and HR trends and challenges

"Enterprise-Proven" is the Prerequisite for Enterprise SaaS Portal Solutions

AT&T Synaptic Storage as a Service. Expand on demand

Trend Micro ranked #1 against real-world malware. Read more.

Webinar: Jump-start your in-house e-discovery with Ringtail QuickCull from FTI Technology

Top Five CIO Challenges

Read the RSA report: Security for Business Innovation

64-page prescriptive guide to security, compliance, and IT operations.

 
 
RESOURCE CENTER