Virtualization's Dirty Little Secrets

The server virtualization drum beat gets louder with Microsoft poised to enter the market, but adoption pitfalls could lead to a bad user experience.

Everyone knows that server virtualization shaves hardware clutter in the datacenter, boosts workloads, brings disaster recovery flexibility, slashes costs and basically saves the planet from nasty carbon emissions. But here's the dirty little secret: Many pitfalls await server virtualization adopters, and a stumble can ruin all your virtual dreams.

More on CIO.com

Adventures in Managing Virtualization

Monster Savings from Virtualization on Sesame Street

Virtualization at Warp Speed: How One Company Made it Fly

The sheer number of potential missteps has Doug Dineley, executive editor of the InfoWorld Test Center, shaking his head. "Virtualization offers irresistible benefits, and also the opportunity to drown."

It can be shocking to suddenly realize that your IT staff is woefully unprepared for virtualization and needs training. Or maybe you'll stumble out of the gate, not knowing that it takes at least a month to get a grip on your server environment. You might be pressed to free up money to cover hidden costs or purchase new equipment -- yes, new servers will likely be needed for what's supposed to be a server consolidation project. Even if you navigate these and other pitfalls, you'll likely be blindsided by virtualization vendors' over-the-top performance claims.

What's Behind the Virtualization Buzz

Server virtualization breaks up the marriage of hardware and software (in this case, between the physical system and operating system software), and thus allows a single physical server to host many virtual servers running different operating systems. The benefits of this basic capability border on computing nirvana, not the least of which is server consolidation. For instance, IBM started moving the workload of its 3,900 servers to 30 virtualized System z9 mainframes running Linux. Big Blue expects to cut energy consumption by 80 percent, or more than $2 million in energy costs. Meanwhile, NetApp consolidated 343 servers to 177 via virtualization and replaced 50 storage systems with 10 new ones.

Indeed, the front lines are awash with server virtualization success stories -- and the drumbeat grows louder every day. EMC's virtualization high-flyer unit, VMware, raised nearly $1 billion in its public offering last summer, based on a highly regarded product (see the Test Center review of VMware Infrastructure 3.0). Citrix Systems, which acquired server virtualization vendor XenSource in December, took the wrappings off of XenServer 4.1 earlier this month. Last week, market researcher Gartner called virtualization "the most important trend for servers through 2012."

Now Microsoft plans to shake up the virtual world with its Hyper-V, a virtual machine manager, or "hypervisor," the company is building into Windows Server 2008. Currently in beta and due out this summer, Hyper-V has already stirred debate among Test Center reviewers. Chief Technologist Tom Yager applauded the offering in February, while Paul Venezia panned Hyper-V in a Test Center preview two months earlier, citing, among other things, that his attempts to run the disk manager often resulted in a lockup. "It has a long way to go to be production-ready," Venezia wrote.

Further, Hyper-V will come to market lacking advanced features, such as live VM migration, that have long been present in VMware's enterprise offering. On the other hand, Hyper-V comes "free" as part of the operating system; and Microsoft's integration of virtual machine management into its pantheon of management tools is sure to be a hit with Windows shops.

Marketing buzz aside, the truth is that server virtualization fundamentally changes the way a datacenter looks and feels -- and no major transformation comes easy.

Gotcha No. 1: You May Not Get the Hardware Savings You Expect

One of the great ironies of server virtualization is that many people expect the technology to save them boatloads of money from the outset when, in fact, it often costs them more. That's because server virtualization demands two things: shared storage and some new servers that are powerful, richly configured, and equipped with hardware memory chips from the likes of AMD and Intel.

Even if you already have these souped-up servers, you're still not out of the woods. Server interoperability issues stymie many virtualization journeys. "You can't mix AMD and Intel platforms together in the same [VMware] ESX cluster," says Chris Wolf, an analyst at the Burton Group. "You cannot move a virtual machine between them without restarting."

The same goes for a storage area network, or SAN. Not every SAN supports a virtualized environment. Also, existing network bandwidth may not be sufficient to handle the needs of a growing number of virtual servers. This means you'll likely end up spending money on new servers, switches, and other tech gear. Even worse, upgrade costs can offset nearly all the initial savings from decommissioned servers, says Matt Prigge, a consultant and Test Center contributor.

When the server virtualization wave began to crest, industry watchers thought that the server market would be in a lot of trouble. After all, virtualization allows people to consolidate many applications onto fewer servers—preferably existing ones. And they were partly right: Gartner believes that virtualization reduced the x86 server market by 4 percent in 2006.

But it soon became apparent, the Test Center's Dineley says, "that you needed to strictly standardize on hardware for your virtual farm." Thus the server market remains strong: Some 8 million servers were shipped worldwide last year, a 6.7 percent increase from the year prior, according to IDC.

Most people tackle hardware standardization and server virtualization slowly, usually when servers are due for retirement. They dabble in noncritical areas such as print servers before moving on to e-mail applications and enterprise databases. "It's a rolling-thunder approach," says John Humphreys, an IDC analyst. "We'll start to see the impact on [server] unit growth two, three, or four years down the road, as more people virtualize."

Gotcha No. 2: Getting the Right Staff Experience is a Challenge

IDG Research Services, a sister unit of InfoWorld, surveyed 464 participants late last year about their virtualization experience. The biggest challenge? Forty-four percent of respondents said inadequate skills and training was the most difficult hurdle, followed by software licensing issues, performance and scalability challenges, and complexity.

So don't expect the IT staff to have all the answers to virtualization from the get-go. It'll take at least a month to gain an accurate understanding of current server workloads, given weekly and monthly spikes, before deciding which servers can be virtualized. In small companies with only a handful of IT folks, you may need to hire -- surprise! -- a pricey consultant to conduct capacity planning.

A small company also may not have the necessary SAN expertise or, for instance, capability to mesh Cisco switches and VMware's complex virtual networking stack. "Virtualization draws together so many different aspects of networking, server configuration, and storage configuration that it requires a well-seasoned jack-of-all-trades to implement successfully in a small environment," Prigge wrote in a "virtual" case study for the Test Center that’s chock full of insight about challenges ranging from pricing and products to technical and skills requirements.

Larger companies don't have it easier, either. Getting a lot of people in disparate teams -- server, storage, business continuity, security -- on the same page is a feat, especially since they traditionally don't talk to each other very much. All of them, though, need to be educated about virtualization. If there's a problem with an application, for instance, an administrator must know where virtual machines exist throughout the server farm so that he doesn't reboot a server and unwittingly take down all the virtual machines on it.

Gotcha No. 3: Performance Boosts Aren't Always What They're Cracked Up to Be

Despite the hard work, virtualization adopters may feel a sting of disappointment. Many will have embraced server virtualization with grand expectations, only to see performance fall short. Burton Group's Wolf points the finger at vendors: "For me, the way VMware advertises performance benchmarks is completely inaccurate."

The vendor publicity materials' virtual machine benchmarks involve running a single virtual machine on a single physical host. But a typical production environment is conservatively eight to 12 virtual machines per physical host. "This paints an overly optimistic picture of performance," Wolf adds. "They also tend to gloss over things like over-allocation of CPU cores" that can tax the hypervisor's CPU scheduler and lower performance.

Memory is another big performance-buster, Wolf says, especially with virtualizing multithreaded applications. When separate threads within an operating system continually try to refresh memory, the hypervisor's shadow page tables get backed up. The result: latency. For applications that rely heavily on memory, latency spikes and application responsiveness deteriorates. Users may start seeing connection timeouts.

"Hardware-assisted memory is one solution, but it's also a crap shoot," Wolf says. "Some applications run better with hardware-assisted memory virtualization while others run better with shadow page tables."

The fallout of lackluster performance can be huge. A company might have to fork out more cash for servers. Business execs may demand that applications be given their own servers again. "Restoring trust in virtualization technology, it may take a couple of years before a company attempts to virtualize again," Wolf says.

And There Are Even More Gotchas to Avoid

Poor performance, unprepared staff, and hidden costs are only a sampling of the pitfalls in server-virtualization adoption. Managing the whereabouts of virtual machines can be a nightmare, given that they can be moved from one physical server to another, or even walk out the door on a portable hard disk. Security risks abound, too. Audit failures due to the lack of full separation of security zones can happen more easily in a virtual environment.

And then there's the threat of virtual server sprawl; new applications are easy to get up and running in a virtual world. "Virtualization increases your appetite for software," IDC's Humphreys says. "One company went from 1,000 applications to almost 1,300." Not only are there potential software licensing costs involved but also the task of tagging and tracking the applications.

Of course, server virtualization's plethora of pitfalls won't stop people from adopting the technology. After all, the benefits in a good implementation can be tremendous. But knowing how to identify and avoid those gotchas can make the journey more pleasant and the reward that much sweeter. "There's no slowing virtualization down; the benefits are too great," Dineley says. "But you will get buried if you don't steer clear of the gotchas."

This story, "Virtualization's Dirty Little Secrets" was originally published by InfoWorld.

Related:

Copyright © 2008 IDG Communications, Inc.

Survey says! Share your insights in our 19th annual State of the CIO study