Solving the ‘Big Application’ Problem, Amazon Style

Ambit Energy

Ambit Energy, which provides gas and electricity to deregulated markets, was founded in 2006.  At the time, John Burke, CIO and co-founder scoured the market for packaged software to run the business, but he didn’t find what he was looking for, so he and his small team built their own.

The ‘Big Application’

When the business started to take off, growing from zero to $325M in three years (today they are $1.5B), Burke saw that they were running almost all of the business – including billing, rating, customer care, and transaction management – on one huge, monolithic system.

“We were growing and expanding into new markets and running the business on one centralized system,” says Burke. “By 2009, it was painful to do deployments.” As such, Burke and his team would plan one day a month for deployments. “The night team would start at midnight and go all night, and then the morning team would come in and repair what was wrong,” he says. “It was hard to replicate a production environment in our test environment because the application was so big and complex.  The process burned out the IT team and frustrated the business.”

Time to Replace Waterfall Application Development

Burke recognized that he had to replace the Waterfall development model his team had been using with something more agile. “Waterfall was such a long pole,” he says. “We would define some new functionality but by the time we deployed it, the business was no longer interested and had already moved on to the next thing.”

It took a little convincing of both his development team and his business partners, but over the next eight months, Burke moved Ambit Energy to an Agile development model. In another major move, he reorganized his department to have nine dedicated software teams to align to the company’s nine business units.

“At first, everyone was very excited about what, to us, was a pretty radical organizational change,” Burke says. “They said, ‘This is great. The business line managers can go directly to their dedicated development teams and tell them what they want.’ It was very empowering to them and they were motivated to make a lot of change.”

But that excitement was short lived when the development teams realized that they had not solved the primary obstacle to driving rapid change. “The development teams were broken into business units, but we still had one large central system, so we were still doing late night pushes,” Burke says. “We had a brief moment of excitement and empowerment, but then we realized that nothing changed. When it came to making system changes, we were in the same boat.”

Taking a Page from Amazon

Around that time, Burke’s senior developers and architects were all talking about Silicon Valley and how Amazon, Netflix and Google were changing their development processes.  “People were talking about an edict that Jeff Bezos gave to his development team at Amazon when they hit their own ‘big application’ problem,” Burke says. “Bezos told them, ‘You need to take your own piece of the large application and rip it out from the rest.  As long as you provide APIs to the large application, you can write your own piece in any language. But your little interfaces have to always be working.”

Despite the fact that Ambit Energy is in a century-old industry, Burke took notice of the Bezos edict.  “I held an open discussion with our development team about whether we could rip our application into smaller pieces,” he says. “There was a small minority of sharp developers who thought it was possible, so we decided to give it a shot.”

Introducing Dev-Ops

Burke and his team renamed the “Software Configuration” team “DevOps”, in part, to herald the directional change for the organization.  “Software Configuration” connoted a hurdle to get code into production, while “DevOps” meant automating and pushing quality code faster. “Our DevOps people were not really DevOps people when we first renamed the team,” says Burke. “But we had to move their mind-set from gatekeeping to facilitating. That was one of the hardest changes we had to make, to get them to see their role as facilitating rapid deployment in pushing code.”

This meant that rather than write the code themselves, the developers now had to configure deployments that were so mathematically correct that they could be all done by scripts and automated software. “That kind of process really makes you think through all of the configurations on your system,” says Burke.

Burke identified a few key architects who believed in the automated continuous delivery model and coached them to bring the vision to the rest of the team.  “Through the influence of these leaders, the people who had been terrified to automate development were starting to get excited about it.”

While Burke was reorganizing and appointing leadership, he still had this large application he had to split up, while the company was growing fast. “We didn’t have an R&D department to figure this out while we performed our day jobs,” Burke says. “We were all working on things that matter to the business, and at the same time, we had to figure out how to break one large system into nine smaller business-aligned systems.”

To break down the application, the IT team had to figure out which automation toolsets to buy and how many extra servers they would need, but to Burke, that was the easy part.  In addition to changing the mindset of the team, the other real challenge was getting buy-in from the business. “It was now 2011, and while the business had gotten excited about the nine dedicated development teams, they started to lose faith when we couldn’t get any deployments out the door. We had to convince our business partners to take the chance and allow us to rip apart the application.”

In the end, after a two-year initiative, Burke and his team did rip apart the application and are now in automated continuous delivery heaven.  “Today, we’re doing roughly 34 deployments a day and we never see a hiccup. Our business teams are on fire because they come to work with a purpose and get things done.”

Burke’s Advice on the Conversion to Automated Continuous Delivery

1. Prepare for heavier network volume:  “Your path to segmented systems will most likely involve a service oriented architecture,” Burke says. “Your network traffic will quadruple because you are moving from strict database calls to multiple network calls. Especially if you manage your own infrastructure, you’ll need to be prepared for some pretty serious volume.”

2. Identify and coach your leaders:  Burke identified a technical evangelist who was so obsessed with continuous delivery that he read about it all the time and practiced on the weekends. Burke gave him the leadership coaching he needed to bring the vision to the rest of the team. “You have to bring your developers and QA people into the discussion; you have to hear from the infrastructure people. If they are gatekeepers who don’t like change, you’ll have trouble getting them into the right mindset. You need an evangelist you can trust who can start demoing the model through prototypes and bring everyone along.”

3. Publically praise your MVPs:  “The magic happened when two teams “got it,” says Burke. They were the first to start using the continuous deployment model and became our Most Valuable Players.  Other developers saw what they are doing and said, “Look at those guys; they are deploying during the day and nothing is disrupted! They are Amazon!”  Burke took that success and broadcasted it with congratulatory emails and public praise.  “You need to come really quickly with one or two wins. Then you market their successes and other teams will take notice. They’ll realize how cool automated continuous delivery is and that they are a part of the next big thing.”

About John Burke

John is the CIO of Ambit Energy, and joined the company in May 2006. He has more than 15 years of leadership and consulting experience in the electric utilities, telecommunications, financial services, venture capital and software development industries. John earned an MBA in Information Systems from The University of Texas at Austin and a Bachelor of Arts in Economics from Rutgers University. He is a member of Phi Beta Kappa and has served as a representative on The University of Texas at Austin Graduate School’s Information Management Steering Committee.

About Ambit Energy

Ambit Energy was founded in 2006 by Jere Thompson, Jr. and Chris Chambless with the goal of providing cost-effective electricity and natural gas services in deregulated markets across the U.S. The services are primarily marketed by more than 250,000 independent consultants via a direct sales channel. Ambit Energy is continually recognized as one of the fastest-growing companies in the retail energy sector, exceeding more than 1 million active customers in 2012. The company is headquartered in Dallas, Texas.