Cloud Testing: the Next Generation
One of the risks of deploying Internet-scale infrastructure and applications is that, until they are put to the test, you can't have 100% confidence that they will scale as expected. Applications and infrastructure that perform well - and correctly - at nominal scale may begin to act wonky as load increases.
Wed, February 09, 2011
Network World — One of the risks of deploying Internet-scale infrastructure and applications is that, until they are put to the test, you can't have 100% confidence that they will scale as expected. Applications and infrastructure that perform well - and correctly - at nominal scale may begin to act wonky as load increases.
Cloud computing and virtualization bring new challenges to testing the scalability of an application deployment. Applications deployed in a cloud environment may be designed to auto-scale "infinitely" which implies you need the same capability in a testing solution.
That's no small trick. Traditionally, organizations would leverage a load testing solution capable of generating enough clients and traffic to push an application and its infrastructure to the limits. But given increases in raw compute power and parallel improvements in capacity and performance of infrastructure solutions, the cost of a tool capable of generating the kind of Internet-scale load necessary is prohibitive.
An internal performance management engineer at F5 Networks (FFIV) applied some math and came up with a staggering $3 million investment required to test an Internet-scale application deployment. That's not feasible for most and not economical. So it seems that testing Internet-scale architectures is going to require Internet-scale solutions - but without the Internet-scale cost.
Cloud computing to the rescue
It's not a huge leap of logic to assume that the same operational model enabling Internet-scalability of applications could do the same - at a fraction of the cost - for Internet-scale testing solutions. All you need is a cloud-deployable load generation client, a couple of cloud computing environments and a way to control the distributed clients to generate the scale necessary to push an application and its infrastructure to the limits.
Experience says this is easier said than done. It's not the deployment that's a problem; it's the management of the distributed test clients. Distribution across multiple providers would prove a nearly insurmountable challenge for most solutions let alone the organization employing them. And you should distribute across providers for several reasons, including:
1. Location of clients matters. Whether it's location-based application logic requiring testing or the reality that applications are not stateless and require client-server affinity, location matters. Combine a narrow range of IP addresses with affinity and scalability challenges are almost certain to appear.
2. Bandwidth. Depending on the cloud provider from which you choose to launch such a test, you may find their network to be the bottleneck. Whether internal or external (to the backbone), launching the scale necessary from a single provider could prove little more than the provider has limited bandwidth or a less than optimal internal network.