Got a crackerjack virtualization pro on your staff? Better keep your eye on that talent.

As virtualization deployments mature from tactical server projects to strategic enterprise initiatives, companies are finding that IT personnel with the necessary skill sets are in short supply. Once trained, virtualization experts can be even more difficult to retain.

"These people are pretty valuable, especially if they have skills in multiple functional areas," says Cameron Haight, a Gartner analyst. "In the '90s, if you were an SAP Basis administrator, you could almost name your price. This is the SAP-Basis admin role of this decade."

The IT department for Georgia's Fulton County has lost two virtualization experts during its ongoing migration from hundreds of single-function x86 servers to blade servers running virtual machines. Both left for a virtualization-related job at another company. Still, Jay Terrell, CTO and deputy IT director, takes the losses in stride. "Sure, we've lost a couple of people, but we've also kept some bright, young talent" by exposing them to virtualization," Terrell says. "If you hold people back, you're going to lose them anyway. I'd rather have people happy and excited while they're here."

Cross-pollination required

Virtualization's reach into nearly every corner of the data center is fueling the talent crunch. Virtualization calls for people who understand how to deal with complex configuration-management, patching and performance monitoring, for example. "Gone are the days of looking only at the running processes on the OS to find out what may be causing a performance problem," says John Turner, director of networks and systems at Brandeis University in Waltham, Mass. "Now you have to look one level up: What are the other resident operating systems doing? What do those loads look like, and how do I optimize those or move them to a box with less load?"

In the bigger picture, virtualization moves networking tasks into the domain of systems engineers, Turner says. "Typically the network engineers would configure network switches, set the appropriate virtual LANs and make sure the network-protection protocols were set up correctly. Now they're handing those off, to a certain extent, to the systems engineers. They're the ones who are setting up the hypervisors, which essentially puts them in the role of having to set up data and storage networking, through this layer of abstraction."

While systems engineers understand switching and routing basics, they generally don't have a grasp of such finer points as loop control, port-channel bonding or packet-sizing, Turner notes. At Brandeis, virtualization has necessitated a higher level of trust among already integrated systems and network teams, as well as a higher skill level. "Our systems engineers are going to be brought up to the level of network engineers. The network engineers are going to be perhaps less responsible for some of the server-side switch-configuration things and more focused on bigger routing issues," he says.