Cloud Performance Metrics: No Standards, So Mileage Varies

Just try to do a fair comparison of how your app runs in two different cloud providers' environments. It's tough now due to a lack of standards.

By Kevin Fogarty
Wed, November 03, 2010

CIO — How well will a particular application perform in a given cloud computing environment? That's a basic question for IT groups who are comparison shopping among cloud optionsand one that's tough to answer at the moment. Apples to apples comparisons prove almost impossible.

Unfortunately, there's not an independent source that can give an individual company a good idea of how well a cloud service will performnor a comprehensive set of tools to help evaluate a service while it's being tested, says Bernard Golden, CEO of consultancy HyperStratus and a CIO.com blogger.

In fact, it's not even easy to compare the costs among cloud providers, let alone measure and map performance, notes Geva Perry, a cloud market strategy blogger.

Wanted: Cloud Benchmarks

In cloud computing there isn't a standard industry benchmark against which to compare all services or even a standard network across which all end users access them, according to a June benchmarking study from Bitcurrent.

Most end-user companies don't have the skills or resources to do that for their own application testing, so they have to retrofit their normal cost- and performance analyses to fit the cloud according to Sean Hackett, research director at The 451 Group.

The Bitcurrent study monitored response time, internal I/O, network performance and other typical metrics and concluded that the services themselves varied so much that the programming requirements, location and internal IT infrastructures of its customers could make results almost too large to be comparable. "Your milage may vary greatly," the report warned.

Within the last year, and especially within the last month, a crop of cloud-measurement services have cropped up to supply that need, with results that range from the ultra-specific to the generic.

"People are using the cloud for a lot of very performance-intensive applicationsVoIP, videoconferencing, high-definition video, disaster recovery, they're finding out ISVs adn carriers are really good at interoperability and connectivity, but can't tell you much about performance," according to Jim Melvin, CEO of network performance-metrics provider Apparent Networks.

"With those types of applications performance is highly dependent on location and the path the traffic takes from your site to theirs. It's not enough to know your bandwidth, there's also latency, jitter and data loss rates to consider," Melvin says.

New Services Try To Fill Gap

This week, Apparent Networks launched a free service designed to give IT managers a detailed look at the average wide-area network performance between two cities.

The service relies on a dozen network-measurement points throughout the U.S. that track network performance to a selected group of cloud providers, including Rackspace, Salesforce.com and other leading brands.

Continue Reading

Our Commenting Policies