Tue, December 02, 2008 — CIO — It's never a good sign when a company identified as a good case-study subject has to put off an interview because of a sudden IT crisis. In the case of Loro Piana USA, the U.S. operation of Italian haute cashmere and wool couturier Loro Piana, S.p.A., the sudden problem had nothing to do with the virtualization, 10-gigabit Ethernet or storage-area networking projects CIO.com had called to talk about.

An electrical circuit-breaker in the company's Stafford Springs, Conn. headquarters tripped and an uninterruptible power supply missed its cue to supply emergency backup power, causing one of the company's two physical servers, and most of its 40 or so virtual servers, to blink out.

"A good chunk of the room went quiet," says Loro Piana USA IT Manager Aaron Martin, who heard the change from his office right outside the server room and came to investigate. "You never want to walk into a server room that's quiet. There's something really wrong when that happens."

"Amazingly," as Martin described it, the other physical server—a normal production server running VMware's ESX virtualization software, including its disaster recovery capabilities&mdash:relaunched all the VMs and applications that had been running on the crashed machine.

In less than 15 minutes all the applications were running again without having lost any data, though the effect was spoiled because the single array that stores all the company's data was on the failed UPS, so none of the file systems were available, Martin says. It was overloaded, so the response time was slow, and the data stores for the five Exchange servers that had been running on the downed server were all corrupted during the crash. Even after rebooting the downed server, switches and the company's primary storage array, which had all been connected to the failed UPS, the Exchange servers were not behaving.

That's one of the major risks of consolidating down to a small number of servers, according to Chris Wolf, analyst at The Burton Group. "I'd be concerned for any organization clustering with just two nodes," Wolf says. "If one went down, you'd have all your VMs on one server and you'll be sweating bullets until you can get that other node replaced or repaired."

Loro Piana actually has other VMware servers running in its New York office—and Martin plans to link the two offices so the company's production servers can double as failover or disaster-recovery targets. But first he has to complete a switchover in telecommunications providers from Verizon to Paetec, which saved so much money on a network connecting 20 stores to the New York and Connecticut offices that Martin was able to upgrade from 1.5 Mbit/sec Multiprotocol Label Switching (MPLS) connections to 20Mbit/sec.

Without that additional speed, and the second data array in the New York office, it's not practical to rely on the two offices to back each other up in real time. The switchover is due to happen the first week in December, however, and the two-way failover link should be functioning soon after, Martin says.

Even so, Wolf says, even many small companies with tight resources find it safer to buy an additional server they can use as a staging area, or development and testing server, and rely on that for disaster recovery. Using a production server could degrade the performance of the servers being pressed into service as backups.

"The issue isn't just getting the servers running again, I'd also worry whether I was meeting my [service level agreements] as well," Wolf says.

A New Storage Array and 10 Gigabit Ethernet Cards

It took a day and a half for Martin's team to rebuild the Exchange data stores and get the servers back online, but the company's other applications, networks and data were all available to the 350 or so Loro Piana workers in the company's Connecticut headquarters, New York office and 20 retail stores around the country.

For Martin, the crash reinforced the need for a stable, replicated, disaster-recovery-enabled network of virtual servers—which he's been building toward for some time, though he actually started serious work on it with a storage array, rather than the virtualization software itself.