Is Your Cloud Provider Exposing Remnants of Your Data?

If your organization uses a multi-tenant managed hosting service or Infrastructure as a Service (IaaS) cloud for some or all of your data—and you aren’t following best practices by encrypting that data—you may be inadvertently exposing it.

Last year, information security consultancy Context Information Security was tasked by a number of its clients, mostly banks and other high-end clients with serious security concerns, to determine whether the cloud was safe enough for their computing needs.

Context studied four providers: Amazon, Rackspace, VPS.net and GigeNET Cloud. And in two of the four providers—and potentially many others—it found a security vulnerability that allowed it to access remnant data left by other customers.

“We were looking at the unallocated portions of the disk,” says Michael Jordan, manager of research and development at Context. “We were able to look through it and started to see there was data in there. That data was hard disk data and it wasn’t our hard disk data.”

Data Remnants Included Personally Identifiable Information

The data Jordan and his team discovered included some personally identifiable information, including parts of customer databases and elements of system information, such as Linux shadow files (containing the system’s password hashes).

Jordan notes that the information wouldn’t be evident to the typical user of cloud servers and would have to be sought. Moreover, he adds, the remnant data was randomly distributed and would not allow a malicious user to target a specific customer. But a malicious user who discovers it could harvest whatever unencrypted data it does contain.

“After examining a brand new provisioned disk on one of the providers, some interesting and unexpected content was discovered,” Jordan and James Forshaw, principal consultant at Context, wrote in a blog post about their discovery. “There were references to an install of WordPress and a MySQL configuration, even though the virtual server had neither installed.

Expecting it to be perhaps just a ‘dirty’ OS image, a second virtual server was created and tested in the same way. Surprisingly, the data was completely different, in this case exposing fragments of a Website’s customer database and of Apache logs which identified the server the data was coming from. This confirmed the data was not from our provisioned server.”

Incorrectly Configured Hypervisors to Blame

The issue, Jordan says, was with the way the providers provisioned new virtual servers and how they allocated new storage space. On the front end, when clients create new virtual servers, they use the provider’s website to select the operating system and amount of storage they require.

On the backend, the provider gathers disk space to contain the virtual image and then overwrites the start of the disk with a preconfigured OS image.

“This means that only the start of the disk is filled with initialized data, as the rest of the disk would never be explicitly written to during provisioning,” Jordan and Forshaw wrote. “If this allocation was being performed using the hosting operating system’s file APIs, this would not normally be a problem. The OS would ensure that any uninitialized data was automatically zeroed before being returned to a user application (or in this case the virtual machine). Clearly in this case it was not using these mechanisms.”

Jordan notes that because the problem lies with the method of configuring hypervisors, it could potentially affect managed hosting providers as well as cloud service providers.

Both providers that exhibited the vulnerability—Rackspace and VPS.net—have since reported that they have patched the vulnerability. Rackspace reportedly worked closely with Context to address the issue, inviting Context researchers to its headquarters and providing access to its engineers, executives and processes. VPS.net uses technology from OnApp, also used by at least 250 other cloud service providers. VPS.net told Context that it rolled out a patch that fixed the issue.

Jordan notes that this issue should not stop companies from using IaaS if there’s a strong business need for it. But he does recommend that customers follow best practices when leveraging the cloud.

“If you are a new customer, you have options,” he says. “You can ensure your data is encrypted when it’s on the hard disk. That way if someone does get access to a portion of a disk, they’ll only see encrypted data.”

Jordan also recommends asking your provider lots of questions about their processes, including how hypervisors are provisioned and deprovisioned. Additionally, he notes that it is the client’s responsibility to harden virtual servers provided by the service provider, and that includes checking out any backdoors providers use to manage the server.

Thor Olavsrud covers IT Security, Open Source, Microsoft Tools and Servers for CIO.com. Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline and on Facebook. Email Thor at tolavsrud@cio.com