by Kevin Fogarty

Server Virtualization: Top Five Security Concerns

May 13, 200910 mins

From VM sprawl to compliance, here's a look at the top security issues worrying IT about virtualized servers. The bad news: Despite improved IT understanding regarding virtualization management, many of the security holes are still ones that companies create themselves.

In surveys of senior-level IT managers, security is consistently one of the top five concerns, along, specifically, with security related to the hot technology of the moment. Most recently those worries have included social-networking technologies such as Twitter and Facebook and other outlets through which employees could turn loose company confidential data. But the security of virtual servers and virtualized infrastructures also rank near the top of the list—and rightly so, according to analysts.

It’s not that virtual servers are any less secure than any other server, according to Neil MacDonald, security and infrastructure analyst at Gartner. In many ways virtual machines are more secure than standalone servers, if only because they are more isolated and because they depend on a single host server, which makes the physical security issue much simpler than if each were on a separate piece of hardware, he says.

“Each one of those virtual servers is still its’ own separate server, though,” MacDonald says. “Each one has its own operating system and configuration that may or may not be according to the standard set by the parent company. And every one of them has to be patched and maintained the same way a non-virtual server does to keep up with potential vulnerabilities; a lot of people forget about that, but it makes the situation a lot more complicated.”

It’s theoretically possible for hackers to attack the hypervisor layer specifically, or to take over a VM and use it to attack other VMs, according to according to Chris Steffen, principal technical architect at Kroll Factual Data, a credit-reporting and financial-information services agency in Loveland, Colo. But this has never happened “in the wild,” so the threat remains theoretical for now. (For more detail on real versus theoretical threats, see’s How to Find and Fix 10 Real Security Threats on Your Virtual Servers.)

“You could also have a virus aimed at the BIOS chip on your machine, but we don’t see too many BIOS viruses, any more,” Steffen says.

The biggest problem with VMs, Steffen and MacDonald say, is the potential for IT or security managers to lose control of them simply by not being able to see the risks as they crop up.

The National Security Agency has taken that concern not only to heart, but to software development labs, coming up with a virtual-server management scheme called NetTop that requires a configuration preventing VMs running on the same machine from interfering with one another. It doesn’t solve all the potential configuration problems, but it does concentrate all the security processes within a specific technology layer and development process.

In 2007, the NSA and contractor General Dynamics expanded that security with a workstation running what it calls the High-Assurance Platform—a virtualized operating system that includes a separate layer of code that is responsible for securing both the virtual operating system and application and the data they use.

Most companies don’t need quite that layer of protection, which was designed for Special Forces groups serving overseas. But they do have a range of pressing security concerns—many of which they either don’t recognize, or don’t appreciate fully, MacDonald says. And that’s the base of the problem.

Here’s a look at the five top virtual server security concerns of the moment.

1. Managing oversight and responsibility

The overarching issue with virtual servers is responsibility, MacDonald says. Unlike physical servers, which are the direct responsibility of the data-center or IT managers in whose physical domain they sit, responsibility for virtual servers is often left up in the air. Should the business-unit that requested it be able to configure and secure it? Should it be the IT manager closest to the physical host? A centralized master sysadmin tasked with management and security for all the virtualized assets in an enterprise?

“People don’t appreciate that when you add virtual servers there’s another layer there of technology in addition to the application and the operating system and the hardware, and you have to secure it, MacDonald says.

2. Patching and maintenance

The most tangible risk that can come out of a lack of responsibility is the failure to keep up with the constant, labor-intensive process of patching, maintaining and securing each virtual server in a company. Unlike the physical servers on which they sit, which are launched and configured by hands-on IT managers who also install the latest patches, virtual machines tend to be launched from server images that may have been created, configured and patched weeks or months before.

Most companies maintain a small number of general-purpose “golden” images from which to launch or relaunch new VMs for many purposes, but also keep dozens or hundreds of server images stored on DVD or disk after being laboriously configured to support specific applications or business requirements, MacDonald says.

“You can take a snapshot of a virtual machine and write it off to disk so you don’t have to recreate it the next time, or for disaster recovery. Just fire off one of these virtual machines sitting in offline libraries. But for the most part they’re not being kept up to date with A/V signatures and patches, ” MacDonald says. “Someone should check when they do launch one, but often they don’t, and there isn’t usually a way to check.”

Both Microsoft and VMware supply patch-management schedules with their base infrastructure products. Both require disk images stored in libraries to be launched periodically so they can be patched.

That’s a tedious process for companies with libraries of hundreds of VM images, however, and does nothing to address the patch status of VMs that are running but might not have been patched or had new antivirus signatures installed for weeks or months. Of course, VMware, HP, and many startup companies are trying to help IT automate much of this work right now with management products.

3. Visibility and compliance

Virtual servers are designed to be, if not invisible, then at least very low profile, at least within the data center. All the storage or bandwidth or floor space or electricity they need comes from the physical server on which they sit. To data-center managers not specifically tasked with monitoring all the minute interactions of the VMs inside each host, a set of virtual servers becomes an invisible network within which there are few controls.

“Virtual switch implementations let the VMs talk to each other, and across the network,” MacDonald says. “But unless you put virtualized security controls—virtual sniffers, virtual firewalls, all the same controls you’d use on a physical server, inside that network, you don’t see what’s going on.”

“There are a lot of compliance and use issues,” McDonald says.”Just because you don’t have a sniffer to see those packets moving between the virtual servers doesn’t mean they’re not there,” MacDonald says. “You could have a HIPPA-controlled workload talking to a non-HIPPA workload, or PCI and non-PCI workloads talking to each other. That puts you in a bad position. You would know if you looked at the packets on that network, but those packets are not coming out of the box for you to look at, so unless you take extra steps, you wouldn’t know.”

Microsoft, VMware and Citrix are all building some level of visibility and control over those interactions into their base products, but the level of function is nowhere near the point that customers will be secure, MacDonald says.

Silicon Valley startup Altor is finding some fans for its virtual firewalls, as is Reflex Systems, which migrated from physical to virtual firewalls to keep up with growth in that market, MacDonald says.

“Cisco’s not there yet, Juniper’s not there; we haven’t reached the tipping point where the traditional networking vendors feel they have to be able to reach into virtual machines,” MacDonald says.

In many cases, customers either don’t know or don’t care about certain risks. A poll of 109 attendees at the RSA Conference 2009 in Las Vegas last month, conducted and published by virtual-security software provider Secure Passage, indicated that 72 percent of respondents have not deployed virtual firewalls of any kind. The most frequent reasons cited: the limited visibility respondents had into virtual networks, the difficulty of managing virtual security and lack of understanding regarding what constitutes a virtual firewall.

VMSafe, the APIs that VMware built into the VSphere version of its virtual infrastructure product, makes it possible for third-party security vendors to apply their applications to VMware VMs. The company also announced at the RSA conference that it had built RSA’s data loss prevention software into vSphere to enhance its security.

“They’re making progress,” MacDonald says of VMware and Microsoft. “They’re not where we need them to be yet.”

Simon Crosby, chief technology officer of Citrix Systems, said during a security debate at the RSA conference that security should be built into the applications, not the hypervisor or virtual-infrastructure management products.

He said paying attention to the security configuration guidelines that Citrix and other hypervisor vendors publish can fix most of the security issues and that industry groups such as the Cloud Security Alliance can extend that guidance to include process-management and policy issues.

4. VM sprawl

Another consequence of the lack of oversight of virtual machines is sprawl—the uncontrolled proliferation of virtual machines launched, and often forgotten, by IT managers, developers or business-unit managers who want extra servers for some specific purpose, and lose track of them later.

VM sprawl wastes resources, creates unmonitored servers that could have access to sensitive data, and sets the company as a whole and IT in particular up for a painful cleanup when a problem crops up later, Steffen says.

“We try to treat the VMs in exactly the same way we do physical machines—with system scans, antivirus, and everything else. That includes going through a procurement process for VMs just as if they were physical machines,” Steffen says.

Forcing business unit managers to fill out requisitions and explain why they want an additional VM, for what, and for how long slows the process down, which could be considered inefficient, but also gives everyone involved time to think about how necessary each new VM is.

“We don’t do that if they need to replace a server they’re already running,” Steffen says. “But with VMs you have the potential for VMs to get completely out of hand and have so many out there you can’t do anything about how secure they are.”

The Secure Passage poll of RSA attendees showed 42 percent were concerned about sprawl, specifically the lack of controls available to keep business unit managers from spawning off new servers at will, rather than coordinating with IT to make sure they are managed and secure.

5. Managing Virtual Appliances

One of the very best things about virtual infrastructures is the ability to buy or test a product from a third-party vendor and have it up and running in minutes, rather than having to clear space on a test server, install the software, get it to talk to the operating system and the network and then, hours later, see whether it does what it’s supposed to, MacDonald says.

Unfortunately, virtual appliances are also virtual pigs in a poke. “There’s an operating system and application in every package, every one with its own configuration and patch status and you have no idea what’s in there or who’s going to maintain it or what the long-term risk is going to be,” MacDonald says. “It has a full application and OS all configured and ready to run. In five minutes you can try out that new anti-spam server. But what OS is in the package and is it patched, and if not, who is going to give you the patch? “

Follow everything from on Twitter @CIOonline