Partners Healthcare CIO John Glaser Faces the Music

All systems go down at some point.

So John Glaser, CIO of Partners Healthcare, wasn’t particularly worried when the electronic medical record (EMR) system used by more than 6,000 doctors and nurses affiliated with Partners started experiencing brief outages in late July. After all, since the start of 2004 the EMR system had experienced anywhere from two to six short disruptions a month—slowdowns or outages lasting from a couple of minutes to several hours. Inconvenient, yes, but not the end of the world. There was no reason to suggest that this was any different.

But the disruptions got worse—the outages occurring with greater frequency. The automated alerts Glaser receives when his systems are strained poured in. Doctors called and e-mailed with complaints. Partners administrators let him know that a lot of people were irritated. In every crisis there is a point at which the notion that this is just a bad week gives way to the recognition that you are treading on thin ice.

By early August, Glaser says, “we realized we were in trouble.”

Between then and mid-September, Partners’ EMR system slowed or shut down 25 times, often for hours at a time. Each disruption affected every doctor on the system; they could not gain access to their patients’ medical records, and at times clinics were forced to turn patients away untreated. The IS department faced heat on all sides—from the doctors whose work was disrupted and from Partners’ administration who feared a medical mutiny over the EMR system.

Dr. Mark Eisenberg, a doctor at Partners’ Charlestown Healthcare Center, couldn’t access his patients’ medical records for 45 minutes during one of these slowdowns. “It is a real problem if we have no record to look at when we see a patient,” Eisenberg says. “There are real concerns about care if you can’t see lab results or what medications someone is taking.”

Partners is Boston’s largest hospital group. The organization includes two of the city’s major academic hospitals—Massachusetts General (MGH) and Brigham and Women’s—as well as smaller community hospitals, clinics and even individual doctors’ offices. The two hospitals have been among the earliest adopters of medical information systems. Some of their doctors have been using the electronic medical records for 15 years, and the overall adoption rate is about 70 percent within Massachusetts General and Brigham and Women’s hospitals. The rest of the community lags far behind, however. Excluding the large hospitals, the adoption rate for the rest of the Partners network is only 10 percent. One of Partners’ top goals over the past two years has been to bring on board these users, a group of approximately 2,000, who are mostly physicians in private practices affiliated with one or more of the hospitals in the Partners system.

EMR systems provide many benefits. In addition to cutting claims and billing costs, they improve patient care by significantly reducing medication or lab test errors that result from sloppy physician handwriting. Such systems also check for drug allergies and adverse interactions with existing prescriptions, and they can advise physicians on what test or procedure is most appropriate.

Even so, they’re a hard sell. Many physician practices have only a handful of doctors—some only one or two—and they don’t always see the financial point of investing in the EMR system. Between networking costs, converting paper records to electronic records, linking or replacing existing office systems and some lost productivity while everyone learns how to use it, installing the system costs somewhere between $5,000 and $10,000, according to Glaser. That’s a lot of money for a small office, and while EMRs will eventually boost productivity, even Glaser admits that there isn’t an ROI. “In our best case it is right on the edge of break even,” he says. “Our hope is that the vision is enough to push it over the edge.”

But every time something goes wrong with the system, that vision of the future moves farther away and Glaser’s job gets harder. The CIO is the first to acknowledge that solving the mystery behind the recurring EMR system slowdowns and restoring the faith of doctors and administrators was a major test of leadership for him. “If [a problem] goes on too long, it erodes the trust that your community has in you,” Glaser says.

No Quick Fix

By the first week of August, it was clear to Glaser that something was wrong with the EMR system. On Friday, Aug. 6, after the system had experienced outages and slowdowns almost every day for two weeks, Glaser realized the status quo was no longer acceptable. He assembled two key groups for a meeting—the people who touch each of the organization’s systems on a daily basis and the IS people who work on the hospital floors.

“I wanted to hear from a technical point of view everything that had happened,” Glaser recalls. “I also asked the people on the front lines, What does it look like for the doctors? How disruptive is this? I needed to know what it feels like to live with this.”

What Glaser heard wasn’t reassuring. The doctors were growing restless. The outages compromised their ability to treat patients. And there wasn’t a quick fix looming. Glaser’s IS staff identified multiple reasons for the disruptions. The largest single culprit: A server that supported the EMR system kept crashing. Other times, it seemed as if the EMR database had outstripped its capacity. Still other times, an unpatched memory leak in Internet Explorer was the cause of the outages.

“It was clear that there was no way we were going to fix this in one week,” Glaser says. “We had to prepare for a series of lousy weeks.”

Glaser quickly came up with a game plan. The technical people would continue trying to diagnose and fix the problem, focusing solely on this task. Other projects, such as compliance with the Health Insurance Portability and Accountability Act, would be put on hold. Glaser would provide cover if anybody asked questions about these temporarily neglected tasks. He also brought in a team of IBM consultants to help diagnose the problem.

In addition, all new EMR system deployments in outpatient physician practices would be put on hold, and a new working group, headed by Glaser’s deputy Mary Finlay, was created to approve all application and infrastructure changes. Glaser, meanwhile, would face the music.

He was the one who had to explain to CEO James Mongan and the rest of the leadership exactly what was going on and what was being done about it. While these administrators made it clear that they were not happy, they didn’t panic, giving the CIO time to fix the problem. Glaser also met with the physician leadership at Partners’ hospitals and explained what was going on. “They don’t want to know what you are doing on a day to day basis,” he says. “But they want to know that you are taking it seriously.”

On Aug. 11, Glaser sent out an e-mail to the entire Partners medical community. In it he acknowledged the recurrent slowdowns and said that fixing the underlying problem was his top priority. He tried to be as frank as possible. “You don’t sugarcoat anything, even if it means that you have to look at yourself in the mirror and not like what you see,” he says in hindsight. “If you screwed up, you need to be able to say so.”

Absorbing the Anger

Glaser knew he had to be visible. “In situations like this, most formal communication has to come from me,” he says. “I have to send this e-mail out; I can’t delegate that. I have to be at the advisory board meeting. I have to meet with the leadership at MGH and the Brigham.”

Sometimes his sole job at a meeting was to get yelled at by doctors. “They are angry and upset, and they want to yell at someone, and it has to be you,” Glaser says. And some doctors took advantage, swearing at him to his face, and accusing him of hurting their patients. “You have to roll with it,” he says. “You have to resist the temptation to fight back.” By absorbing the anger himself, Glaser allowed his employees to solve the problems without interruption.

As it turned out, almost all of the disruptions could be traced to the same root cause. In its effort to expand the reach of the EMR system, Partners had neglected to upgrade its operating system, and the old system just couldn’t handle the load it was forced to bear. The IBM consultants confirmed this. The old system hadn’t been designed to support a record system as large and dispersed as Partners’.

Upgrading the system, however, would take time, primarily because the vendor, Intersystems, had never worked with an application infrastructure as complex as Partners’, and every piece of the new system would have to be tested. In the meantime, Glaser’s team spent September making short-term fixes, such as adding more servers. By October, there were only four incidents, and by the end of December, the new upgraded system was in place. Glaser hopes the new operating system will keep the EMR database up 99.9 percent of the time. In the aftermath of the crisis, Glaser has had to reassess his plans to extend the EMR capabilities to all of Partners’ affiliated doctors. That ambitious project will have to wait. “If you continue to roll out the implementation, you just look reckless, like you are so bent on the goal that you are ignoring reality,” he says.

Instead, Glaser will focus on reestablishing his department’s track record. “If you have a good reputation when you go into [a crisis], people believe that you are honest and that you can fix it,” he says. “Your words have an aura of credibility. But if a problem goes on too long, history becomes irrelevant.”

Glaser won’t know for months the extent to which the prolonged disruptions have affected the credibility of his IT leadership. But he doesn’t have time to worry about the repercussions now. He’s focused on keeping the EMR system up and running 24/7, because that, after all, is the future of medicine.