All systems go down at some point.
So John Glaser, CIO of Partners Healthcare, wasn’t particularly worried when the electronic medical record (EMR)
system used by more than 6,000 doctors and nurses affiliated with Partners started experiencing brief outages in late
July. After all, since the start of 2004 the EMR system had experienced anywhere from two to six short disruptions a
month—slowdowns or outages lasting from a couple of minutes to several hours. Inconvenient, yes, but not the end
of the world. There was no reason to suggest that this was any different.
But the disruptions got worse—the outages occurring with greater frequency. The automated alerts Glaser receives
when his systems are strained poured in. Doctors called and e-mailed with complaints. Partners administrators let him know
that a lot of people were irritated. In every crisis there is a point at which the notion that this is just a bad week
gives way to the recognition that you are treading on thin ice.
By early August, Glaser says, “we realized we were in trouble.”
Between then and mid-September, Partners’ EMR system slowed or shut down 25 times, often for hours at a time. Each
disruption affected every doctor on the system; they could not gain access to their patients’ medical records, and at
times clinics were forced to turn patients away untreated. The IS department faced heat on all sides—from the
doctors whose work was disrupted and from Partners’ administration who feared a medical mutiny over the EMR system.
Dr. Mark Eisenberg, a doctor at Partners’ Charlestown Healthcare Center, couldn’t access his patients’ medical
records for 45 minutes during one of these slowdowns. “It is a real problem if we have no record to look at when we
see a patient,” Eisenberg says. “There are real concerns about care if you can’t see lab results or what medications
someone is taking.”
Partners is Boston’s largest hospital group. The organization includes two of the city’s major academic
hospitals—Massachusetts General (MGH) and Brigham and Women’s—as well as smaller community hospitals, clinics
and even individual doctors’ offices. The two hospitals have been among the earliest adopters of medical information
systems. Some of their doctors have been using the electronic medical records for 15 years, and the overall adoption
rate is about 70 percent within Massachusetts General and Brigham and Women’s hospitals. The rest of the community
lags far behind, however. Excluding the large hospitals, the adoption rate for the rest of the Partners network is
only 10 percent. One of Partners’ top goals over the past two years has been to bring on board these users, a group
of approximately 2,000, who are mostly physicians in private practices affiliated with one or more of the hospitals
in the Partners system.
EMR systems provide many benefits. In addition to cutting claims and billing costs, they improve patient care by
significantly reducing medication or lab test errors that result from sloppy physician handwriting. Such systems also
check for drug allergies and adverse interactions with existing prescriptions, and they can advise physicians on what
test or procedure is most appropriate.
Even so, they’re a hard sell. Many physician practices have only a handful of doctors—some only one or two—and
they don’t always see the financial point of investing in the EMR system. Between networking costs, converting paper
records to electronic records, linking or replacing existing office systems and some lost productivity while everyone
learns how to use it, installing the system costs somewhere between $5,000 and $10,000, according to Glaser. That’s a lot
of money for a small office, and while EMRs will eventually boost productivity, even Glaser admits that there isn’t an
ROI. “In our best case it is right on the edge of break even,” he says. “Our hope is that the vision is enough to push
it over the edge.”
But every time something goes wrong with the system, that vision of the future moves farther away and Glaser’s job
gets harder. The CIO is the first to acknowledge that solving the mystery behind the recurring EMR system slowdowns
and restoring the faith of doctors and administrators was a major test of leadership for him. “If [a problem] goes on
too long, it erodes the trust that your community has in you,” Glaser says.
No Quick Fix
By the first week of August, it was clear to Glaser that something was wrong with the EMR system. On Friday, Aug. 6,
after the system had experienced outages and slowdowns almost every day for two weeks, Glaser realized the status quo
was no longer acceptable. He assembled two key groups for a meeting—the people who touch each of the organization’s
systems on a daily basis and the IS people who work on the hospital floors.
“I wanted to hear from a technical point of view everything that had happened,” Glaser recalls. “I also asked the
people on the front lines, What does it look like for the doctors? How disruptive is this? I needed to know what it
feels like to live with this.”
What Glaser heard wasn’t reassuring. The doctors were growing restless. The outages compromised their ability to
treat patients. And there wasn’t a quick fix looming. Glaser’s IS staff identified multiple reasons for the disruptions.
The largest single culprit: A server that supported the EMR system kept crashing. Other times, it seemed as if the EMR
database had outstripped its capacity. Still other times, an unpatched memory leak in Internet Explorer was the cause
of the outages.
“It was clear that there was no way we were going to fix this in one week,” Glaser says. “We had to prepare for a
series of lousy weeks.”
Glaser quickly came up with a game plan. The technical people would continue trying to diagnose and fix the problem,
focusing solely on this task. Other projects, such as compliance with the Health Insurance Portability and Accountability
Act, would be put on hold. Glaser would provide cover if anybody asked questions about these temporarily neglected tasks.
He also brought in a team of IBM consultants to help diagnose the problem.
In addition, all new EMR system deployments in outpatient physician practices would be put on hold, and a new working
group, headed by Glaser’s deputy Mary Finlay, was created to approve all application and infrastructure changes. Glaser,
meanwhile, would face the music.
He was the one who had to explain to CEO James Mongan and the rest of the leadership exactly what was going on and
what was being done about it. While these administrators made it clear that they were not happy, they didn’t panic,
giving the CIO time to fix the problem. Glaser also met with the physician leadership at Partners’ hospitals and explained
what was going on. “They don’t want to know what you are doing on a day to day basis,” he says. “But they want to know
that you are taking it seriously.”
On Aug. 11, Glaser sent out an e-mail to the entire Partners medical community. In it he acknowledged the recurrent
slowdowns and said that fixing the underlying problem was his top priority. He tried to be as frank as possible. “You
don’t sugarcoat anything, even if it means that you have to look at yourself in the mirror and not like what you see,”
he says in hindsight. “If you screwed up, you need to be able to say so.”
Absorbing the Anger
Glaser knew he had to be visible. “In situations like this, most formal communication has to come from me,” he says.
“I have to send this e-mail out; I can’t delegate that. I have to be at the advisory board meeting. I have to meet with
the leadership at MGH and the Brigham.”
Sometimes his sole job at a meeting was to get yelled at by doctors. “They are angry and upset, and they want to yell
at someone, and it has to be you,” Glaser says. And some doctors took advantage, swearing at him to his face, and accusing
him of hurting their patients. “You have to roll with it,” he says. “You have to resist the temptation to fight back.” By
absorbing the anger himself, Glaser allowed his employees to solve the problems without interruption.
As it turned out, almost all of the disruptions could be traced to the same root cause. In its effort to expand the
reach of the EMR system, Partners had neglected to upgrade its operating system, and the old system just couldn’t handle
the load it was forced to bear. The IBM consultants confirmed this. The old system hadn’t been designed to support a
record system as large and dispersed as Partners’.
Upgrading the system, however, would take time, primarily because the vendor, Intersystems, had never worked with an
application infrastructure as complex as Partners’, and every piece of the new system would have to be tested. In the
meantime, Glaser’s team spent September making short-term fixes, such as adding more servers. By October, there were
only four incidents, and by the end of December, the new upgraded system was in place. Glaser hopes the new operating
system will keep the EMR database up 99.9 percent of the time. In the aftermath of the crisis, Glaser has had to reassess
his plans to extend the EMR capabilities to all of Partners’ affiliated doctors. That ambitious project will have to
wait. “If you continue to roll out the implementation, you just look reckless, like you are so bent on the goal that you
are ignoring reality,” he says.
Instead, Glaser will focus on reestablishing his department’s track record. “If you have a good reputation when you
go into [a crisis], people believe that you are honest and that you can fix it,” he says. “Your words have an aura of
credibility. But if a problem goes on too long, history becomes irrelevant.”
Glaser won’t know for months the extent to which the prolonged disruptions have affected the credibility of his IT
leadership. But he doesn’t have time to worry about the repercussions now. He’s focused on keeping the EMR system up
and running 24/7, because that, after all, is the future of medicine.