Halamka on Beth Israel's Health-Care IT Disaster
"We depend upon the network, but we also take it for granted," Sands says. "It’s a credit to [Halamka] that we operate with a mind-set that the computers never go down. And that we put more and more critical demands on the systems. Then there’s a disaster. And you turn around and say, Oh my God."
Halamka had become an ad hoc communications officer for anyone looking for information. Halamka was the hub of a wheel with spokes coming in to him from everywhere?the CAP team, executive staff, clinicians and the outlying hospitals. Halamka leaned on his emergency room training at the Harbor-UCLA Medical Center in Los Angeles, during the height of gang violence in the ’90s. Rule one: Stay calm and friendly.
"But I’ll be honest, 48 hours into this, with no sleep, the network’s still flapping, I had a brave face on, but I was feeling the effects," Halamka recalls. "I was feeling the limitations of being a human being. You need sleep, downtime. You need to think about shifts, or humans will despair."
He found himself dealing with logistics that had never occurred to him: Where do we get beds for a 100-person crisis team? How do we feed everyone? He improvised.
"You don’t know the details you’re missing in your disaster recovery plan until you’re dealing with a disaster," he says. For example, the paper plan was, in essence, the Y2K plan. Besides the fact that it was dated, it didn’t address this kind of disaster.
Recovery plans are usually centered on lost data or having backups for lost data, or the integrity of data. At Beth Israel Deaconess, the data was intact. It was just inaccessible.
That led to Halamka’s chief revelation: You can’t treat your network like a utility.
"I was focusing on the data center. And storage growth. After 9/11, it was backup and continuance. We took the plumbing for granted. We manage the life cycle of PCs, but who thinks about the life cycle of a switch?"
This is a valuable insight. Networks indeed have gotten less attention than applications. But at the same time, Callisma’s Rusch says, he hadn’t seen a network as archaic as Beth Israel’s in several years. "Many have already gotten away from that 1996 all-switched model," he says. "There are probably a couple of others like this out there."
Others agree with Rusch’s assessment. "I think the danger is people start thinking the whole health-care IT industry is flawed and a train wreck waiting to happen," says the CIO of another hospital. "It’s not. We all watched the heroic effort they made over there, but we’re not standing around the water cooler talking about how nervous we are this will happen to us. We’ve had these issues. They scared us enough a few years ago that we took care of the architecture problem."





