Adventures in Managing Virtualization

Evan Jafa, CIO of First American , says getting your virtualized servers set up and running right really is just the start of any IT leader’s virtualization work. And that if you don’t think holistically about virtualization, you’re in for a rude surprise.

MORE ON VIRTUALIZATION

Survey: Your Virtualized State in 2008

Monster Savings from Virtualization on Sesame Street

Virtualization at Warp Speed

“I think lots of folks are going to find out that virtualization poses a whole new set of concerns for security, networks and applications,” says Jafa.

In fact, he calls management a “make or break issue” for virtualization. “Monitoring and management becomes absolutely critical in the virtual environment,” says Jafa, who started work about three years ago on a large project to consolidate First American’s data centers and standardize its storage and networking technologies.

To that end, his organization is directing much of its energy now at management processes and tools, wide application of ITIL process to the virtualized environment, and even reshaping the IT staff.

Jafa hopes that in the future virtualization will play into better management of his networking technologies as well. He’s already testing some new technologies from vendors including Cisco to advance that goal. If you’re trying to do holistic virtualization planning, Jafa’s advice may prove particularly interesting.

Big Provisioning and DR Wins

First American’s business, which posted $8.2 billion in 2007 revenues, delivers information such as mortgage, real estate and financial data to clients in those industries and others. So while many IT leaders closely monitor virtualization projects to ensure that application service levels don’t slow down for internal users, Jafa’s team also keenly focuses on application performance for external customers.

The company today has two data center locations, one in California and another in Texas, with about 4,500 physical servers, soon to be 6,500 (due to newly added business units.) It serves 40,000 to 50,000 concurrent users during peak business hours; its network carries 10 to 21 terabytes of data per hour, averaging about 15 terabytes per hour, Jafa says.

To keep all that data flowing smoothly, Jafa says his tool chest currently includes offerings from VMware, Microsoft, BEA (WebLogic) and “lots of Linux”. His team is also using HP’s SiteScope tools for hardware performance monitoring.

Jafa says he defines virtualization as “achieving the highest efficiency of any given system at any give time”. In servers, virtualization is netting him 50 to 60 percent utilization rates, compared to 10 to 15 percent utilization rates on standard servers, he says. The team also sees a 30 to 35 percent reduction in energy consumption per virtual machine compared to its physical counterpart, Jafa notes.

To date, IT has virtualized 25 to 26 percent of First American’s servers, Jafa says. These include many production servers, plus test and development and staging boxes. In some cases, whole business units now have had all their servers virtualized, he says. “My goal is to reach 50 percent virtualized this year,” he adds.

“Any server that IT’s using is going virtual,” Jafa says, which means some 300 servers will be consolidated onto seven or eight physical machines. “We continue to work on virtualizing disaster recovery.” Jafa is bringing disaster recovery in house: “That’s going to be completely virtualized”, he says, noting he hopes to complete the DR project by end of year.

Virtualization has also helped Jafa’s team as the company consolidates and centralizes IT services for an array of business units. “The process of bringing them on and migrating them has become faster,” Jafa says, noting that the company’s IT provisioning has improved by a factor of 65 percent, changing from weeks to days.

On the disaster recovery and high availability front, virtualization has given Jafa equally dramatic results. For every 100 servers, he now puts aside four servers for failover purposes and six to eight servers for workload help at peak usage time for the company’s customers. Previously, the company’s high availability arrangement meant that he needed to put aside 200 servers for the same purpose.

Politics and Process Lessons

But server virtualization is the start of the changes at First American, not the end. “As we started to think about virtualization, we started to think about standardization,” says Jafa, as in standardizing processes and utilizing shared resources, across not only servers but also storage and networking technology.

In the course of doing his planning, he’s run across two key challenges IT leaders cited in CIO’s recent survey, Your Virtualized State in 2008: Workload balancing and IT staff management issues.

“Capacity planning has become critical in the virtual world,” Jafa says. “That’s one we’re still maturing and figuring out. We’re using some of the HP [SiteScope] tools for capacity planning.”

On the staff issue, he says he hasn’t had to deal with much squabbling among IT members who are being asked to work together in new ways, noting his team is mostly excited about virtualization. The bigger political challenge, he says, involved First American’s diverse collection of business groups. “We have many business units and to get them off the notion of ‘my servers’ and ‘my storage’ took a lot of work,” Jafa says. “I credit our (IT) change management organization.”

Still, he says, he does face a wave of organizational questions, ones that he feels most IT leaders will also have to battle.

“Virtualization will start posing a serious challenge organizationally to IT,” he says. “The current vertical disciplines that we have will not be appropriate.” For example, he says, IT may need a horizontal group with members from various teams to handle provisioning.

How is his group dealing with this situation? “So far we haven’t torn down the vertical organizations,” Jafa says. “Through the architecture organization, we have brought those teams together.” His IT team now has an overall architecture team that works on design, architecture and process questions, he says.

Process, he believes is key. Jafa, who was already a big fan of using ITIL to standardize IT process before he started the virtualization efforts, weaves ITL into his team’s efforts in the physical and virtual worlds. Jafa says close to 55 percent of his team is ITIL-certified and he has a fulltime ITIL instructor in house.

“In the virtual world, processes become super critical,” Jafa says. “We want to make sure to test our processes to see how they will do. For example, provisioning has to be constantly defined,” he says.

“We just did a whole assessment of exactly what will the impact of virtualization be on our processes,” he says. “We have to meld some of our processes for the virtual world,” he says, noting network design and even software design has to change in the virtualized environment. “Now we can look at the long term impact.”

On that networking front, Jafa is now working with Cisco to test out technologies such as Virtual Switching Systems (VSS) and VFrame (a family of provisioning and automation tools), in a quest to make his network more standardized and simpler to manage.

“Network provisioning becomes as critical as server provisioning,” Jafa says. “We are very interested in virtualization of the LAN, so we don’t have all these protocols for storage and fibre in the network. The simpler the protocols and the more collapsed the protocols are, the easier the virtualized world will be managed.”

For more on what that means, see “The Virtualized Network of the Future.”

The Virtualized Network of the Future

“Virtualization of the LAN” is a concept that can mean different things to different people. The basic idea is the same as server virtualization: Take a single physical resource, like a LAN, and virtualize it in terms of resource sharing and simplified management.

If you have many separate LANs, you can use virtualization to consolidate them into one switch, notes Deepak Munjal, marketing manager of Cisco Data Center Solutions. You can also consolidate network services, such as firewalls and network load balancing. (For more details on this, see Cisco’s description of its Virtual Switching System, or VSS.)

Looking ahead, vendors such as Cisco would like IT leaders to think about virtualizing the actual cables that come out of a server. Today, various cables including fibre channel interfaces to storage and Ethernet interfaces to management tools come out of physical servers. “You can start using new tools to collapse those protocols,” onto two cables, a large link and one for redundancy, says Cisco’s Munjal.

This is the idea behind Fibre Channel over Ethernet, an open industry standard that makes its Cisco debut in the recently released Nexus 5000, a 10 Gigabit Ethernet top-of-rack switch.

In Cisco’s vision of the future, the benefits in a network design using switches like this one resemble the benefits of sever virtualization: less hardware to manage, reduced power and better utilization of hardware asset, and hopefully, simplified administration.

Keep in mind, Cisco just shipped that Nexus 5000 switch in early April and even forward-thinking IT leaders like Jafa are still in early test mode with such technologies, so it may be a while before you see large enterprise data centers actually revamping their network design.