Inside One CIO’s Storage Nightmare

As Kindred Healthcare CIO Rick Chapman begins to discuss storage strategies, he asks a revealing rhetorical question.

MORE ON CIO.com

How to Do Storage Virtualization Right

Rethink Your Storage Infrastructure to Save Money

HP Aims to Simplify Midsize Company Life with Storage Array

“The question you should ask me is, Why would I be talking about this, as opposed to the VP of data center operations?” The simple answer, he explains, is that Kindred’s infrastructure and storage costs sit high atop his agenda—and his executive committee’s—right now. As wave after wave of new data flows in—electronic patient records, e-mails, insurance and billing files, and government-mandated documentation—Chapman feels the squeeze on the overall cost and performance of his IT operations, he says.

Storage is suddenly in the C-suite spotlight.

“All of a sudden, what used to be a mundane technology component and what the executive committee or the board wouldn’t have cared much about and would have trusted to me and my subordinates, now we have to be more transparent, and be able to defend the purchases and the cost structure of the infrastructure,” Chapman says.

“That used to not be the case,” he says. “But now I have to make sure we’re as cost effective and still able to provide the business service reliability for the company.”

Chapman is no small-time CIO, nor is he one of those CIOs more comfortable in the back room than in the boardroom. Kindred Healthcare is the largest for-profit long-term healthcare provider in the United States with $4 billion in revenues, 52,000 employees and 600 facilities in 39 states. Chapman is not only the CIO but the chief administration officer, and he sits on the executive committee. He has worked in top IT spots at healthcare giants Columbia HCA and Humana in the past.

So Chapman knows the complexities of health care from both the business and IT sides, and what he sees on the IT side of late leaves him in awe. “We’re growing [data volumes] by 40 percent a year, and now we have over 400 terabytes—just unimaginable volumes of data from what we had just a few years ago,” Chapman says. “We’ve seen exponential growth, and it keeps pressure on the storage platforms.”

As a result, he’s vowed to eliminate traditional tape backup systems entirely in the next few years.

The ‘Save Everything’ Mentality

Chapman and other CIOs in health care face a vicious cycle of mandatory document retention that feeds on itself. Government regulations, such as the Health Insurance Portability and Accountability Act (HIPAA), force every organization and every person to save every electronic medical record and e-mail for longer and longer periods of time.

The 2008 Healthcare Information and Management Systems Society (HIMSS) survey reflects CIOs’ concern about the topic. In looking at 2007 and 2008 survey data, “government regulation and compliance issues” showed the greatest year-over-year increase among all categories when CIOs and IT leaders were asked to choose the business issue that would most impact health care during the next two years.

In addition, healthcare institutions now face pressure from their legal departments to be able to produce all documents that relate to malpractice, and Medicare fraud and abuse cases. “This has caused us to have to keep all e-mail forever, and other documents that are critical, because they’re subpoenable and discoverable in legal cases,” says Chapman. “Saying that we deleted them is not a good reason anymore.”

The sum total of all this storage could be overwhelming for many healthcare organizations. A Frost & Sullivan “Healthcare Storage Report” predicts that by 2010, medical organizations will have to hold nearly 1 billion terabytes of data, which is roughly the equivalent of 2 trillion file cabinets’ worth of data.

Of course, in this “save everything” environment, storage gets expensive and resource-intensive. “It’s mind-boggling and completely nonproductive,” Chapman notes. “It’s that hidden cost of health care that you don’t see at times.”

Tales of the Tape

One critical piece of this unwieldy storage puzzle is that Kindred needs fast and easy access to backup, archiving and disaster recovery systems. For example, Kindred Healthcare often receives subpoenas to provide “select e-mails from executives who were having conversations about a certain topic during a certain time period,” Chapman says. “All of this is kind of recent, and all this has ratcheted up in the last five years as more [legal and government] attention has come to this industry.”

Like many companies, Kindred has historically relied upon tape backup systems and massive, centralized data libraries for archiving and recovery. Tape, however, has its limits, as storage volume grows. In turn, the nightly batch processes, or once-a-day backup, are nearly impossible. “It takes more time to back up and recover than we have time available in the data center overnight,” Chapman says.

Concerns regarding tape’s limitations are widespread among healthcare CIOs. John Halamka, the CIO of Beth Israel Deaconess Medical Center (BIDMC) and Harvard Medical School, writes on his blog that storage backup and data recovery is at the top of his list of things that will keep him awake in 2008.

“Tape backup, which has been in use at BIDMC for decades, suffers from a variety of problems,” Halamka writes. “Tape backups are time-consuming. Tapes are fragile and require physical security when transported. The time required to retrieve and recover from tape stresses our service availability objectives.”

In a recent InfoStor Magazine survey of storage professionals, nearly one-third of respondents rated their inability to meet storage-restore objectives as the greatest vulnerability of their data protection strategy.

Both Halamka and Chapman are moving to new storage technologies to help solve their tape problems. “It’s a matter of survival,” Chapman says.

Kindred is implementing virtual tape library, or VTL, technology, which is an appliance that uses data storage virtualization technology to improve compression on disks. A key facet of VTL is data “deduplication,” an approach that reduces a big portion of redundant data that can hog disk space and provides quicker access to data, as compared to tape. For example, each time a document is backed up, only the changes that a user makes to that document are saved—not the entire document or copies of the document.

VTL technology, Chapman notes, is proving to be more cost-effective for Kindred than storage area network (SAN) services from big providers such as EMC. “[VTL] is low-profit stuff,” he says, “and they want you to stay on their high-profit-margin storage devices.”

Chapman is using VTL technology from vendor Sepaton, and Halamka is rolling out similar technology from Data Domain, which he describes in his blog. Both of them plan to rid their organizations’ reliance on tape as soon as possible. Halamka wants to completely eliminate the need for tape in his data center in two years. And Chapman says: “In five years we don’t want any tape.”

Repurposing Storage Dollars

“Doing more with less” has become a familiar refrain for almost every business today, but this is especially true for health care. In the 2008 HIMSS survey, respondents said that the most significant barrier to successful implementation of IT was (for the eighth year in a row) a lack of financial support or budget.

At Kindred Healthcare, Chapman says he’s on a mission to “fund more business initiatives out of reducing the base IT infrastructure costs.”

“We can find money out of our existing spend to fund more work on frontline business initiatives, like application development,” he says. This newfound agility, as he calls it, will give IT more credibility and help grow the business.

One example: Chapman is taking the wealth of historical data that Kindred maintains (such as patient, billing and financial information) and providing it to his users for business analysis. Powerful data warehouse analytic tools “allow people to scan this massive data and convert it into actionable information to deal with a business issue,” Chapman says. “That’s been how we literally pay for what we have to spend on storage, because there’s big value in that.”

Last year, for instance, the Medicare Payment Advisory Commission (MedPAC) had proposed a “very onerous” rate restructuring and reimbursement plan for long-term care hospitals, Chapman says. Kindred Healthcare and other similar organizations had 60 to 90 days to react to the new regulations and give their input on why this was or was not a good plan. (It was not a good plan for Kindred, he says.)

“In a couple of days, working with our reimbursement departments, we could model that new reimbursement [plan] and apply it to the entire previous year’s activity in the hospital division, and come out with what the impact would be,” Chapman says. That information, mined from Kindred’s systems, was then given to Kindred’s government relations representative, who ended up giving testimony before the House Ways and Means committee.

In the end, the proposal was altered, saving Kindred “50 percent of what was going to be a giant cut in reimbursement,” Chapman says. “Not every example is as dramatic as that one was, but it’s that kind of direct ‘connect the dots’ to an outcome that affects the bottom line of the company.”