All systems go down at some point. So John Glaser, CIO of Partners Healthcare, wasn't particularly worried when the electronic medical record (EMR)\nsystem used by more than 6,000 doctors and nurses affiliated with Partners started experiencing brief outages in late\nJuly. After all, since the start of 2004 the EMR system had experienced anywhere from two to six short disruptions a\nmonth\u2014slowdowns or outages lasting from a couple of minutes to several hours. Inconvenient, yes, but not the end\nof the world. There was no reason to suggest that this was any different. But the disruptions got worse\u2014the outages occurring with greater frequency. The automated alerts Glaser receives\nwhen his systems are strained poured in. Doctors called and e-mailed with complaints. Partners administrators let him know\nthat a lot of people were irritated. In every crisis there is a point at which the notion that this is just a bad week\ngives way to the recognition that you are treading on thin ice. By early August, Glaser says, "we realized we were in trouble."Between then and mid-September, Partners' EMR system slowed or shut down 25 times, often for hours at a time. Each\ndisruption affected every doctor on the system; they could not gain access to their patients' medical records, and at\ntimes clinics were forced to turn patients away untreated. The IS department faced heat on all sides\u2014from the\ndoctors whose work was disrupted and from Partners' administration who feared a medical mutiny over the EMR system.\nDr. Mark Eisenberg, a doctor at Partners' Charlestown Healthcare Center, couldn't access his patients' medical\nrecords for 45 minutes during one of these slowdowns. "It is a real problem if we have no record to look at when we\nsee a patient," Eisenberg says. "There are real concerns about care if you can't see lab results or what medications\nsomeone is taking."Partners is Boston's largest hospital group. The organization includes two of the city's major academic\nhospitals\u2014Massachusetts General (MGH) and Brigham and Women's\u2014as well as smaller community hospitals, clinics\nand even individual doctors' offices. The two hospitals have been among the earliest adopters of medical information\nsystems. Some of their doctors have been using the electronic medical records for 15 years, and the overall adoption\nrate is about 70 percent within Massachusetts General and Brigham and Women's hospitals. The rest of the community\nlags far behind, however. Excluding the large hospitals, the adoption rate for the rest of the Partners network is\nonly 10 percent. One of Partners' top goals over the past two years has been to bring on board these users, a group\nof approximately 2,000, who are mostly physicians in private practices affiliated with one or more of the hospitals\nin the Partners system. EMR systems provide many benefits. In addition to cutting claims and billing costs, they improve patient care by\nsignificantly reducing medication or lab test errors that result from sloppy physician handwriting. Such systems also\ncheck for drug allergies and adverse interactions with existing prescriptions, and they can advise physicians on what\ntest or procedure is most appropriate. Even so, they're a hard sell. Many physician practices have only a handful of doctors\u2014some only one or two\u2014and\nthey don't always see the financial point of investing in the EMR system. Between networking costs, converting paper\nrecords to electronic records, linking or replacing existing office systems and some lost productivity while everyone\nlearns how to use it, installing the system costs somewhere between $5,000 and $10,000, according to Glaser. That's a lot\nof money for a small office, and while EMRs will eventually boost productivity, even Glaser admits that there isn't an\nROI. "In our best case it is right on the edge of break even," he says. "Our hope is that the vision is enough to push\nit over the edge." But every time something goes wrong with the system, that vision of the future moves farther away and Glaser's job\ngets harder. The CIO is the first to acknowledge that solving the mystery behind the recurring EMR system slowdowns\nand restoring the faith of doctors and administrators was a major test of leadership for him. "If [a problem] goes on\ntoo long, it erodes the trust that your community has in you," Glaser says.No Quick FixBy the first week of August, it was clear to Glaser that something was wrong with the EMR system. On Friday, Aug. 6,\nafter the system had experienced outages and slowdowns almost every day for two weeks, Glaser realized the status quo\nwas no longer acceptable. He assembled two key groups for a meeting\u2014the people who touch each of the organization's\nsystems on a daily basis and the IS people who work on the hospital floors."I wanted to hear from a technical point of view everything that had happened," Glaser recalls. "I also asked the\npeople on the front lines, What does it look like for the doctors? How disruptive is this? I needed to know what it\nfeels like to live with this."What Glaser heard wasn't reassuring. The doctors were growing restless. The outages compromised their ability to\ntreat patients. And there wasn't a quick fix looming. Glaser's IS staff identified multiple reasons for the disruptions.\nThe largest single culprit: A server that supported the EMR system kept crashing. Other times, it seemed as if the EMR\ndatabase had outstripped its capacity. Still other times, an unpatched memory leak in Internet Explorer was the cause\nof the outages. "It was clear that there was no way we were going to fix this in one week," Glaser says. "We had to prepare for a\nseries of lousy weeks." Glaser quickly came up with a game plan. The technical people would continue trying to diagnose and fix the problem,\nfocusing solely on this task. Other projects, such as compliance with the Health Insurance Portability and Accountability\nAct, would be put on hold. Glaser would provide cover if anybody asked questions about these temporarily neglected tasks.\nHe also brought in a team of IBM consultants to help diagnose the problem.In addition, all new EMR system deployments in outpatient physician practices would be put on hold, and a new working\ngroup, headed by Glaser's deputy Mary Finlay, was created to approve all application and infrastructure changes. Glaser,\nmeanwhile, would face the music. He was the one who had to explain to CEO James Mongan and the rest of the leadership exactly what was going on and\nwhat was being done about it. While these administrators made it clear that they were not happy, they didn't panic,\ngiving the CIO time to fix the problem. Glaser also met with the physician leadership at Partners' hospitals and explained\nwhat was going on. "They don't want to know what you are doing on a day to day basis," he says. "But they want to know\nthat you are taking it seriously."On Aug. 11, Glaser sent out an e-mail to the entire Partners medical community. In it he acknowledged the recurrent\nslowdowns and said that fixing the underlying problem was his top priority. He tried to be as frank as possible. "You\ndon't sugarcoat anything, even if it means that you have to look at yourself in the mirror and not like what you see,"\nhe says in hindsight. "If you screwed up, you need to be able to say so."Absorbing the AngerGlaser knew he had to be visible. "In situations like this, most formal communication has to come from me," he says.\n"I have to send this e-mail out; I can't delegate that. I have to be at the advisory board meeting. I have to meet with\nthe leadership at MGH and the Brigham." Sometimes his sole job at a meeting was to get yelled at by doctors. "They are angry and upset, and they want to yell\nat someone, and it has to be you," Glaser says. And some doctors took advantage, swearing at him to his face, and accusing\nhim of hurting their patients. "You have to roll with it," he says. "You have to resist the temptation to fight back." By\nabsorbing the anger himself, Glaser allowed his employees to solve the problems without interruption.As it turned out, almost all of the disruptions could be traced to the same root cause. In its effort to expand the\nreach of the EMR system, Partners had neglected to upgrade its operating system, and the old system just couldn't handle\nthe load it was forced to bear. The IBM consultants confirmed this. The old system hadn't been designed to support a\nrecord system as large and dispersed as Partners'. Upgrading the system, however, would take time, primarily because the vendor, Intersystems, had never worked with an\napplication infrastructure as complex as Partners', and every piece of the new system would have to be tested. In the\nmeantime, Glaser's team spent September making short-term fixes, such as adding more servers. By October, there were\nonly four incidents, and by the end of December, the new upgraded system was in place. Glaser hopes the new operating\nsystem will keep the EMR database up 99.9 percent of the time. In the aftermath of the crisis, Glaser has had to reassess\nhis plans to extend the EMR capabilities to all of Partners' affiliated doctors. That ambitious project will have to\nwait. "If you continue to roll out the implementation, you just look reckless, like you are so bent on the goal that you\nare ignoring reality," he says.Instead, Glaser will focus on reestablishing his department's track record. "If you have a good reputation when you\ngo into [a crisis], people believe that you are honest and that you can fix it," he says. "Your words have an aura of\ncredibility. But if a problem goes on too long, history becomes irrelevant."Glaser won't know for months the extent to which the prolonged disruptions have affected the credibility of his IT\nleadership. But he doesn't have time to worry about the repercussions now. He's focused on keeping the EMR system up\nand running 24\/7, because that, after all, is the future of medicine.