Amazon's Data Center Outage Reads Like a Thriller

When an Amazon Web Services data center lost power early Wednesday, the details about the event trickled out over Amazon's operations status board. Performance monitoring firms offer services designed to get ahead of such problems.

By Patrick Thibodeau
Fri, December 11, 2009

Computerworld — When an Amazon Web Services data center lost power early Wednesday, the company wrote about the unfolding event with the brevity and tension of one its bestselling pot boilers.

Data Center Definitions and Solutions

Our anonymous author, who we'll call Sysadmin, begins his story simply, without emotional complications and love interests.

"We are investigating connectivity issues for instances in the US-EAST-1 region," Sysadmin writes on Amazon's operations status board at 1:08 a.m. PT.

With one sentence, we're intrigued. Something's up with Amazon's data center in Northern Virginia, just a short drive to Washington; Tom Clancy country.

You can almost feel what's going on. Cloud-based services are crashing and there's a scramble for answers. Elsewhere, PC screens are refreshed as readers wait for an update from Sysadmin, (Kindle edition not yet available). Some 18 minutes pass. Tension builds.

Sysadmin offers an update, referring to isolated "power issues."

Inside the data center a real, red-light-flashing drama unfolds.

At first, a "single component of the redundant power distribution system failed in this zone," Sysadmin would later write in a postscript for his audience. But while the data center staff worked on that component, there was a twist: "A second component, used to assure redundant power paths, failed as well."

Customers are losing connectivity.

Whether data center staff cheered when the problem was fixed remains a mystery. But as soon as the "defective power distribution units were bypassed, servers restarted and instances began to come online shortly thereafter," wrote Sysadmin.

Readers wouldn't get those details until later, when Sysadmin had more information and time. In those early minutes of the outage, only essential information gets to anxious readers. At 1:51 a.m., Sysadmin wrote: "The underlying power issue has been addressed. Instances have begun to recover."

At 2:11 a.m., he writes again: a recovery is well under way.

All that's left are the reviews. That's where companies like Wellesley Mills, Mass.-based Apparent Networks Inc. come in.

In November, Apparent Networks launched its Cloud Performance Center , an online service that allows anyone to review -- in real-time -- the performance of 16 cloud providers, including Amazon and Google . It covers such things as bandwidth capacity, latency and data loss, then scores them overall.

Jim Melvin, president of the privately held Apparent Networks, said his firm can continuously monitor network performance over WANs using technology it has extended to the cloud. The monitoring is done with a "very lightweight stream of packets" that continuously travels the network to monitor activity and cloud performance.

Continue Reading

Originally published on www.computerworld.com. Click here to read the original story.
This whitepaper offers a detailed look into the fundamentals of HP NonStop SQL solutions. See how this system delivers unprecedented levels of application availability with fail-safe data integrity and meets the needs of enterprises with large-scale business critical applications.
Learn how your answer to this question compares to your peers by taking this quick poll. See how your peers are dealing with the challenge of ensuring a highly capable server infrastructure as technological shifts impact the application server platform.
With increasing data growth, comes increased need for data security.  The existing DLP model, with a focus on compliance/enforcement is not sufficient as the data discovery and classification capabilities are not granular enough.  Read this paper to find how you can efficiently and accurately manage your risk by rapidly inventorying and classifying your data and then developing remediation workflows that support business needs. 
This paper breaks down attack sources into four categories: external, malicious insiders, accidental insiders, and unknown.
The rapid growth of data and technology is creating challenges for organizations as this digital data is considered to be business communications and must be preserved according the same industry-specific regulations governing the retention and discovery of emails and more traditional forms of electronic communications. This paper examines the role that Data Loss Prevention ("DLP") technology can play in helping organizations address the challenges of locating information in response to electronic discovery.
This research, conducted by the Ponemon Institute, focuses on issues relating to the use of data protection solutions such as endpoint encryption and data loss prevention within the workplace.
As greater numbers of datacenter servers transition from the physical to the virtual world, the components of virtualization success come to the fore. What scores of organizations have discovered is that success is derived from an optimal pairing of the right software platform with the right hardware platform.
Have you been looking to hear about customer's experiences with the new VMware vCenter Site Recovery Manager product? View this webcast to learn about VMware customer, Navicure, and their experiences testing and evaluating the recovery manager, their progress in implementing it in their environment and their advice other customers considering using vCenter.
Many enterprises have discovered that the use of virtualization to support desktop workloads creates a range of significant benefits. These benefits include price efficiencies, improved IT management and greater agility and choice for end users.

This VMware sponsored webcast with IDC will provide both quantitative measurement of the business value -- defined as the expected ROI -- and qualitative analysis associated with the use of VMware View™. IDC will also provide an analysis of the View Composer and ThinApp™ features of VMware View, including the business value of these solutions and an overview of how they work.

Attend this webcast to learn about:
- Challenges and barriers that might impede the adoption of desktop virtualization
- Navigating roadblocks to facilitate a strategic implementation
- Optimizing qualitative and quantitative benefits to IT and your business
VMware recently announced VMware vFabric™ Data Director, a new database deployment and operations platform that enables enterprise IT organizations to offer database as a private cloud service. Built on top of VMware vSphere 5, vFabric Data Director enables IT organizations to ontrol database sprawl through automation and consistent policy enforcement and accelerate application development cycles with self-service database management. Attend this webcast to learn how vFabric Data Director can help you build database-as-a-service in your datacenter.
A simple, cost-effective disaster-recovery solution for virtual environments is high on the agenda for IT organizations as they virtualize more business-critical applications with VMware. VMware vCenter™ Site Recovery Manager-the market-leading disaster-recovery product-ensures the simplest and most reliable disaster protection for all virtualized applications. VMware vCenter Site Recovery Manager provides centralized management of recovery plans, enables nondisruptive testing and automates site-failover processes.
Traditional disaster recovery solutions are often too expensive, complex and unreliable to meet business requirements. As a result, IT departments are hesitant to expand disaster protection beyond their most critical applications, largely because they are uncertain whether the quality of the protection is really worth its cost. VMware vCenter™ Site Recovery Manager 5 is the market-leading disaster recovery product that addresses this situation for organizations of all kinds. It complements VMware vSphere to ensure the simplest and most reliable disaster protection for all virtualized applications.
Newsletter Sign-Up »

Receive the latest news test, reviews and trends on your favorite technology topics

Choose a newsletter
  1. View all Newsletters | Privacy Policy
Sponsored Links
Resource Center