Halamka retreated to his office late Friday night. He lay down on the floor, pager in one hand, cell phone in the other, and fell asleep for the first time in two days. Two hours later, his cell phone rang.

Saturday Helplessly Hoping

Half awake, Halamka heard a staffer tell him they had found two more spanning tree errors, one at a facility called Research North and one in cardiology. Both had eight hops, one too many. They planned to cut the redundant links and move the traffic to the core network.

No one knew for sure how severely this would tax poor Libby030 and its counterparts. The team decided to build a redundant core with routing infrastructure as a contingency plan that would bring CareGroup out of 1996 and into 2002 in terms of its network.

At 8 a.m., two more Cisco 6509 routers (with switching capabilities) arrived from San Jose. Three hours before that, a trio of Cisco engineers from Raleigh, N.C., landed in Boston. They spent all day building a redundant network core.

Sands felt uncomfortable doing rounds that morning. "Patients sort of expect you to know their histories," he says. "But without that dashboard of information I’d get from the computer, I had to walk up to patients I had treated before and ask basic questions like, What allergies do you have? Even if I thought I remembered, I didn’t trust my memory. It was embarrassing, and I was worried."

Progress on the network was slow. No one wanted to suggest that the current tack?building the redundant network core while severing the redundant links?was definitely the answer. At 9 a.m., Halamka met with senior management, including CareGroup CEO Paul Levy. "I can’t tell you when we’ll be functioning again," Halamka confessed.

Admitting helplessness is not part of Halamka’s profile. "You never catch John saying, I’m scared, or I messed up," says one of his peers from the Health Data Consortium. "This had to be hard for him."

"When John told us he couldn’t tell us when we’d be up, we stopped having him as part of our twice-a-day reports," Epstein recalls. The intent was to free Halamka from his communications duties so that he could focus on the problem. But Epstein was also becoming frustrated. He recalls thinking that "we didn’t want to keep sending out memos to the staff that said, Just kidding, we’re still down."

"If I had felt, in the heat of the battle, that someone could have done a better job than me, if I felt like I was a lesion, then I would have stepped aside," Halamka says. "At no time did I think this, and at no time was I fearful for my job. Am I personally accountable for this whole episode? Sure. Absolutely. Does that cause emotional despair? Sure. But I had to fix it."