Netflix Releases Customized Amazon Control Console
Netflix's home-built open source tool manages Amazon workloads
Tue, June 26, 2012
IDG News Service (New York Bureau) — One of the biggest users of Amazon's cloud services, Netflix has released an open source console called Asgard that it developed to manage its Amazon deployments.
A home-built console was necessary for Netflix, given the amount of work it conducts on Amazon Web Services (AWS).
"For everyday large-scale operations, the AWS Management Console has not yet met the needs of the Netflix cloud usage model, so we built Asgard instead," wrote Joe Sondow, the Netflix software engineer for engineering tools who led development of the application.
Amazon's own management tools have a number of shortcomings, Sondow said. For instance, Amazon offers no way to automate the workflow around deploying an application. It does not provide a log of recent administrative actions. And it doesn't provide many ways to integrate in-house engineering tools, such as Jenkins.
Sondow built the application to address these issues, using the Grails framework. Named after the home of the Norse god of thunder and lightning, Asgard allows administrators to manage their Amazon cloud deployments from two perspectives unavailable on Amazon's own management interfaces, that of applications and clusters.
For Netflix, running an application may require multiple Amazon services, including those for load balancing, auto-scaling, launching and security. Asgard provides a single view for managing these services, organized around each application. Asgard maintains an application registry in Amazon SimpleDB, which keeps track of the many resources supporting a single application. It can also show how long each instance of the application has been running and when it was created.
Asgard has a similar interface for managing clusters, one that lists the individual nodes of each cluster. The software can remove individual nodes, a practice called rollback, if they prove to be troublesome. It also can manage a rolling push, in which all the nodes of a cluster are replaced one by one, without disrupting the operation of the cluster as a whole.
Netflix' use of Amazon is expansive. Except for billing and a number of other functions, "We run all of Netflix out of EC2," explained Jeremy Edberg, the lead cloud reliability engineer at Netflix, during a talk at the Qcon developer conference last week in New York City. The company had to use a public cloud because "we couldn't build data centers fast enough," he said. Netflix has 25 million customers who stream movies and televisions shows from the service.
To help its developers quickly create new features and services, Netflix has built an entire platform from CentOS and Ubuntu virtual servers running on Amazon. "We essentially built a PaaS," Edberg said. Developers can easily build Java or Python applications that will run automatically on the platform, which uses the Apache Tomcat, or another application server of the developer's choosing.
As for Asgard, "this a great tool that helps us be very quick and agile in our deployments and management of Amazon," Edberg said. "We are really hoping people will push patches back, because there is stuff that it doesn't yet do."
The code, available on GitHub, is available under a Apache License, Version 2.0.
Amazon itself appears to be pleased with Asgard. Amazon Chief Technology Officer Werner Vogels praised the technology in a short message posted on Twitter.