Business Continuity Lessons from the Blackout of 2003

When the lights went out Aug. 14, darkening much of the Northeast from Detroit to New York and into Canada, Matthew Hunt was ready. The CTO of Omnipod, a communications services provider based in a post World War II-era Manhattan building, had a week’s worth of diesel fuel to run a power generator in the basement. Several blocks away at Capital Printing Systems, CIO Craig Sisler activated his company’s backup data site in New Jersey as his colleagues draped plastic over the servers in case the building’s sprinkler system rained on them.

Business continuity means preparing for the unexpected. This time, the unexpected was the biggest power outage in American history, affecting up to 50 million people. And this time, CIOs at companies big and small—some prodded by memories of September 11, others dusting off Y2K to-do lists—deployed continuity plans that, experts say, effectively limited the blackout’s damage.

But even where recovery efforts went swimmingly, experts and IT executives say there are lessons to learn from the blackout of 2003.

Put people first. Employees at all levels of a company need to know not only what’s happening, but what they need to do. Sisler says he’s making communications a priority as he updates his company’s disaster recovery plan. “We need to set up a voice-mail system at a separate geographic location that employees can call to get a recorded message. Without that, we left a lot of people literally and figuratively in the dark,” he says.

Protect systems and data integrity. The blackout is inspiring many companies to implement new backup and storage procedures so that “when the power comes back on, there’s an ability to resynchronize data and systems,” says Michael Croy, business continuity solutions director at Forsythe Solutions Group. Ken Steinhardt, technology analysis director at storage vendor EMC, adds that most storage systems come with backup battery supplies (like EMC’s, which automatically store cached data onto a disk when the power goes out). The largest companies run remote data replication systems to second and sometimes third data centers.

Prioritize business processes. “You need to make sure you can recover those processes that are most critical,” Croy says. For example, a bank that’s a Forsythe client realized that when the lights go out what people want is cash. So the bank needs to prioritize getting its ATMs back online.

Practice, practice, practice. “We did a ton of testing on procedures during Y2K preparations,” says Jim Simmons, CEO of SunGard Availability Services. “And when I woke on Jan. 1, 2000, I thought, boy, what a waste that exercise was. Now, three years later, it’s come in handy.” SunGard saw 67 companies demand backup and recovery services during the blackout, its second-highest total (after 9/11, 77 customers declared disasters).

Postmortems should be in the plan. That bank now knows to focus on ATMs. And Sisler sought feedback from all 105 staffers at Capital Printing. “You can become insular when you have a committee working on something,” he says. One item he’s investigating is food. “It may sound crazy, but I contacted an organization that distributes MREs (meals ready-to-eat) to the military. MREs have a long shelf life, and they don’t take up much space.”