Server Benchmarks: Have They Outlived Their Usefulness?

When you consider what IT veterans love most, benchmarks rank right up there. But in today’s world of wildly-varying virtualized server setups and highly-customized application deployments, it’s time to ask: Are server benchmarks an early love that you should let go? IT has been conditioned since the 1980’s to measure all sorts of hardware and software buying decisions using performance benchmarks. But some of the current discussion around CPU, server and virtualization benchmarks makes me shake my head. And I’m not the only one, as I learned when I met this week with AMD’s Kevin Knox, VP of commercial business.

“Even one or two years ago, customers focused on benchmarks,” when comparing enterprise servers, he told me. “Now they focus more on power, cooling, density, environmental issues.”

Some people will say “AMD is trying to de-emphasize benchmarks? No surprise.” That’s because a long time ago, in the early 90’s, Intel was winning the benchmark competition for desktop PC microprocessors. Ancient history, if you ask me. I covered microprocessors in the late 90’s, as Intel and AMD raced to the one-gigahertz mark for desktop processors. They both arrived at that finish line at roughly the same time, at about the same level of performance for desktop PCs. Since then, both companies’ chips have become more than powerful enough for desktop PC applications, and PC buyers are much more concerned with choosing a PC vendor than a chip vendor. 

Servers, of course, are a different story. Enterprise IT buyers of servers have far more complex needs than desktop PC buyers. Today’s servers, with multiple chips with multiple CPU cores, continue to be performance benchmarked heavily, by vendors, industry groups and technology publications.

But now, a growing number of AMD-based server customers, Knox says, are going beyond traditional server performance benchmarks. These enterprise IT leaders are creating their own equations for comparing servers, using factors such as (you guessed it) power, cooling, and other items that affect real life in a data center, he says. 

Knox calls it the “build your own equation” approach.

I find this fascinating, for a couple of reasons. First, technology journalists have spent man hours that would make NASA engineers cringe trying to come up with desktop and server buying-criteria equations similar to this, equations which reflect more than just performance scores on CPU tasks, application tasks, and the like. It’s incredibly hard. 

Second, CIOs in some of the most demanding data center environments (such as financial services) are building these individualized buying decision equations, Knox says.

Maybe they’ve decided the same thing that I have after reading early accounts of vendors trying to out-benchmark each other on server virtualization and other early attempts to do virtualization benchmarking: It’s not going to be meaningful to server buyers, anytime soon.  

As I have learned while interviewing many CIOs in the past year regarding virtualization, every virtualized server is different. When you talk about measuring the performance of a physical server full of VMs, there are a ton of variables in what that server looks like, even just comparing the servers inside one company. Many more variables arise when you compare the servers of various companies.

I think it’s too hard to paint a picture of a typical server with a typical blend of VMs with a typical blend of apps inside those VMs, which you could then benchmark and make widely meaningful. Therefore, I don’t find server virtualization benchmarking terribly useful yet.  

And it seems, neither do these CIOs who are building their own how-to-choose-a-server equations.

Even the question of how many VMs you can safely run on one physical machine is highly complex. The answer depends on the apps themselves, for instance, whether they’re demanding I/O performance apps like databases, or Web server apps, or e-commerce server apps. The “how many VMs?” question also may depend on your attached storage and, let’s not forget, your personal appetite for risk. Some IT departments run mission-critical applications inside VMs; other IT vets would not dream of it yet, and use VMs only for lower-profile apps.  

Make no mistake, virtualization and server benchmarks and buying decisions will be intricately tied from now on; if a large number of your servers aren’t running VMs now, they will be within the next few years.

And make no mistake, as Microsoft, Citrix, Oracle and others put the pressure on VMware in the server virtualization market, some of these vendors will bring up confusing benchmarks to try to win over customers. It’s up to you IT pros to tell the vendors if you find those benchmarks perplexing or irrelevant to your daily concerns.   

CIOs care about many measures with regards to their data centers full of servers running virtual machines: utilization rates, rack space requirements, and cooling costs are just three examples of criteria that traditional performance benchmarks couldn’t address well. But a build-your-own equation could. And, a build-your-own equation could be highly personalized, to reflect, let’s say, your keen interest in cooling, while another CIO may want to weight utilization rates as more important.

I can’t build that equation for you, no matter how much I know about virtualization. You know your own data center’s real-life demands, your in-house applications, and your comfort level with risk much better than I do, and much better than a team of benchmarkers in a lab will ever know. Your data center today is a very individualized picture.  

This means that build-your-own equations for judging servers may vary widely.

As AMD’s Knox puts it, “It’s not going to be one size fits all.” 

So now, I’d like to hear from some of you IT pioneers building these equations. What made you decide to build the server-buying equations? How hard is it? Who on your team is helping? What are the challenges? What have you learned so far? How long have you been using this approach? How is it helping your business? I’m sure many other IT departments would be interested to learn from your early experiences. Please drop me an e-mail.

As for the rest of you, what do you think? Can you dump your love of performance benchmarks when it comes to servers?  Do you wish you could build your own server buying equation? Let’s hear about it.