Hawaii Earthquake Provides Disaster Recovery Lessons

CIO’s Margaret Locher reports on disaster-recovery efforts immediately after the recent earthquake in Hawaii.

Ron Uno, manager of information management at Kuakini Health Systems in Honolulu, Hawaii, discovered that a glitch in his emergency generator prevented his disaster recovery system from kicking in when an earthquake hitting 6.7 on the Richter scale struck the islands at 7:30 Sunday morning, October 15.

The way the recovery system was supposed to work was this: When the main center in the basement of one of the physician tower buildings loses power, the system is supposed to switch over to a secondary unit located in the main hospital.

But that didn’t happen.

The patient care system, including Electronic Medical Records and all patient information, went down from the time the quake hit until 4 p.m. Sunday. Uno and his team at Kuakini have been working fairly successfully to switch Kuakini to a paperless environment (Uno says they’re 50 percent there), so when the power went out, he “freaked out.” And the fact that the island’s entire infrastructure went down, including land lines and cell phones, didn’t help matters. (Uno ended up driving in to the hospital to see what he could do.)

The hospital’s generator didn’t kick in when the power went out (Uno doesn’t yet know why), but the main data center’s generator did. However, the connection from the data center into the hospital failed, even though the servers were up. Consequently, doctors couldn’t access patient information, which prevented them from providing anything but the most rudimentary care.

The total downtime was about eight hours. “Way too long,” Uno says.

“[The disaster] revealed a lot of areas that we can improve on,” says Uno. “People talk about recovery for data centers but it’s not only that, it’s for everything.”

The outages caused by the earthquake taught Uno a lot about his infrastructure. “You don’t know where you are vulnerable until something happens to you.” Uno’s next step will be to put together a report on what happened and what can be done to prevent the system from failing again.

For more on this subject, visit CSOonline’s ABCs of Business Continuity and Disaster Recovery Planning.