Last summer when Wendy Cebula was shopping for a new vehicle, energy efficiency and lower emissions topped her list of requirements, along with four-wheel drive (her family lives on a hill). Cebula, then CIO at VistaPrint, a $152 million online supplier of custom print services, eventually chose a hybrid model instead of the traditional SUV. Even though she didn’t think the incremental savings on gas would make up for the higher sticker price, she says “it was the right thing to do as a human being.”
But as a corporate executive, Cebula, now VistaPrint’s COO, can’t lead with her heart. The right thing to do is whatever enables the business to grow. Those decisions come down to dollars and sense, not what’s best for the planet. But every now and then, the two converge.
In late 2005, Cebula noticed her data center’s costs were growing. The company, whose operations are almost completely automated, was adding 100,000 customers a month—and growing at nearly 60 percent a year. There was no sign the demand for data necessary to serve those customers would stabilize. When Cebula and Aaron Branham, VistaPrint’s vice president of technology and operations, dug into data center operations in detail, they discovered that energy costs were rising significantly.
If there were some way to lower power costs or increase energy efficiency, they could cut expenses. To Cebula, an avid recycler who tries to impart environmental awareness to her kids, doing so would be a double win. She could do “what’s good for the environment and what’s good for the bottom line,” she explains.
Until recently, the environmental impact of the data center was largely ignored. But today, energy experts estimate that data centers gobble up somewhere between 1.5 percent and 3 percent of all electricity generated in the United States. At the top of the range, that’s about the amount of electricity it takes to power the entire state of Michigan for a year. Market research company IDC (a sister company to CIO’s publisher) estimates that companies spent $26.1 billion to power and cool servers worldwide in 2005. That’s more than was spent to power all the commercial buildings in 17 states—from Delaware to Florida and west to Texas, according to the Department of Energy’s most recent energy consumption survey.
And that’s not all. According to the Uptime Institute, a consortium of companies devoted to maximizing efficiency and uptime in the data center, more than 60 percent of the power used to cool equipment in the data center is completely wasted. In fact, notes a recent study by the group, energy costs have replaced real estate as the primary data center expense. “Data centers that used to cost $10 million now cost $100 million,” says Jonathan Koomey, staff scientist at Lawrence Berkeley National Laboratory. “That kind of expenditure gets C-level attention.”
It’s garnered government interest too. A federal law enacted in December compels the U.S. Environmental Protection Agency to examine power consumption in data centers, evaluate what technology manufacturers are doing to increase energy efficiency and determine what incentives could convince companies to adopt more energy-efficient technology. The European Union is studying the level of carbon emissions from computer equipment. Down the road, local and federal governments in the United States and abroad may end up penalizing organizations that operate inefficient data centers, according to Rakesh Kumar, Gartner research vice president.
The combination of financial, environmental and legislative pressure will force IT organizations to develop greener data centers, says Kumar. By 2011, Gartner predicts, a quarter of new data centers will be designed for maximum energy efficiency and minimum negative environmental impact. But what that means may vary by organization. “There’s no generally accepted, standardized way to build a green data center,” says Kumar.
At VistaPrint, becoming green has proven easier than Cebula thought. The company bought more energy-efficient servers and improved utilization in its primary data center in Bermuda, steps that have reduced energy usage by 75 percent. As a result, the company expects to save nearly half a million dollars over three years and estimates it will reduce its output of carbon dioxide emissions by several hundred tons in this year alone. That’s equivalent to taking more than 100 cars off the road for a year.
VistaPrint also decided to locate a new data center in Canada, where hydroelectric power—a renewable energy source—keeps power costs stable and has potential to lower VistaPrint’s electricity bills by another 70 percent. “We were able to reduce our footprint at a time when it was very important to us financially,” says Cebula. “And it’s much more green.”
High-Density Power Surge
Back in 2000, when VistaPrint founder and CEO Robert Keane moved company headquarters from Paris to Waltham, Mass., the fledgling operation was bringing in just over $6 million in annual sales (the company makes business cards and other printed products to order). During the next five years, the company (which is officially based in Bermuda) saw its revenue explode. VistaPrint outgrew its 7,000 square feet of suburban Boston office space and moved a few miles north to a Lexington, Mass., location eight times larger.
During those years, the company focused on automating and optimizing the product design and manufacturing process at its Venlo, Netherlands, plant. The only concern when purchasing equipment—whether for the company’s main Bermuda data center (which runs VistaPrint’s website and transaction systems and is hosted by Cable & Wireless) or for the one in Lexington (which supports internal systems and IT production)—was that the equipment work. As a result, VistaPrint procured a hodgepodge of servers: back-end systems, front-end systems and databases each ran on different gear, says Cebula. By 2004, VistaPrint got caught up in the blade craze, purchasing machines from IBM.
In the fall of 2005, shortly before Cebula became CIO, VistaPrint’s then-COO, Alex Schowtka, hired Branham as director of IT operations. Branham thought blades were a mistake. The problem was the total power pull. “Blade servers look great on paper,” Branham says, “but you start piling them into a rack and suddenly you’re out of power, you’re out of AC.”
VistaPrint wasn’t the only company that was getting burned by its decision to buy high-density equipment. With quality data center space priced at a premium, companies sought out more compact gear, and vendors happily obliged. “The focus was on getting as much computer power in as small a package as possible,” explains Gartner’s Kumar. “Energy was not part of the design mentality.”
But blade servers need more power than less-dense hardware. A full rack of high-density servers requires 20 to 30 kilowatts of electricity, while traditional data centers are designed to provide 2 to 3 kilowatts per rack. Meanwhile, according to the Uptime Institute, using high-density equipment triples or quadruples facility cooling costs. Energy prices have risen as well.
CIOs who assumed that data center costs would decline as servers got cheaper and more powerful received a rude awakening. In what Kenneth Brill, founder and executive director of the Uptime Institute, calls “the meltdown of Moore’s Law,” the energy required to power and cool $1,000 worth of server equipment has skyrocketed from 8 watts in 2000 to 109 watts today—eroding some of the benefits from more powerful chips. In the best-case scenario, says Brill, within five years it could take 157 watts to run the same $1,000 worth of hardware; in the worst case, 1,650 watts. In fact, the blade server’s selling point—its size—has become irrelevant for some customers. Data center operators find themselves using only two servers in a rack designed for 10; packing any more than two high-density servers in a rack makes them too hard to cool.
That was VistaPrint’s quandary. It was the first Cable & Wireless customer in the Bermuda data center to install blades. And the outsourcer was none too thrilled. Cable & Wireless wanted to limit VistaPrint to one blade server per rack. VistaPrint got the vendor to agree to placing two servers per rack, but talk started of an energy surcharge. For years, most outsourcers charged for data center space based on square footage—a metric that did not take into account electricity costs. As a result, says Lawrence Berkeley’s Koomey, the outsourcers created a perverse incentive. “If you charge by the square foot, of course the customers want fully packed racks,” Koomey says.
A Virtual Solution
For Branham, joining VistaPrint in September 2005 was like “a trip back in time.” He had spent the eight previous years at Monster.com and had seen data center operations expand from one server to 1,000. He had spent the past year working on server virtualization. But Branham saw bigger challenges at VistaPrint. “From afar it looks like a print company. But once you get inside and see what’s really going on, it’s really a technology company,” Branham says, noting that everything from sales to manufacturing to shipping is run largely on custom-built software. “At Monster,” says Branham, “we weren’t anywhere near this data-focused.”
Like Monster in earlier days, VistaPrint wasn’t making the most efficient use of its data center. “They picked technologies that were right at the time, not right for where they were going.” Branham knew that virtualization would reduce power consumption. But Cebula knew that she wouldn’t convince anyone to spend more money on servers without a solid business case. So Branham ran the numbers, focusing on power expenditures.
Calculating the total energy consumption of a server isn’t straightforward. How much power a server consumes depends on how you use it. “It’s not something that our vendors talk about a lot,” says Cebula. The vendors were, nevertheless, able to provide data when asked. Using that information, Branham made the case for a greener data center crystal clear.
Branham calculated that if VistaPrint continued to use blades, this solution (conceived of as four racks housing eight IBM blade servers) would eat up approximately 32,000 watts (4,000 per server) and require 9.1 tons of air conditioning to cool. He estimated that the alternative, using eight HP Proliant DL 585 rack-mounted servers and 110 VMware instances, would require 5,500 watts (50 watts per virtual machine) and just 1.6 tons of AC. In addition, virtualization would enable VistaPrint to make better use of its CPU capacity. As is the case in many companies, VistaPrint was wasting additional energy by using only 20 percent of its server capacity. In a 24-hour period, the typical x86 server is used to only 5 percent or 10 percent of its capacity, says Gartner’s Kumar, while energy consumption even in idle mode is 60 percent to 80 percent of the energy consumed in use.
But the development team was concerned about stability and performance in a virtual environment. A year earlier, the IT group had attempted virtualization on the blades. Running VMware on the high-density hardware had been a bust. Memory limitations on the blades limited performance and the number of instances that could be run on each server. So when Branham used the “V”-word again, the development group got nervous. Branham won them over with a pilot project that proved the performance case. Swapping in the new servers in the Bermuda data center would mean eating the investment VistaPrint had made in the blades. But Cebula and the executive management team couldn’t argue with the ROI. Over three years, the greener solution would save VistaPrint $450,000—more than the cost of the hardware refresh.
Cleaner, Cheaper Electricity
By the fall of 2006, competition over parking in VistaPrint’s Lexington lot led its leaders to conclude it was time to move again, to a bigger space across the street.