Robinhood is an online trading platform that is on a mission to democratize finance and make it possible for everyone to participate in the financial system. The company has experienced explosive growth in the past year, the growing pains that go with it, and some important lessons along the way. Adam Wolff spent two years as the VP of Engineering. He is also a member of ENG, the peer-network of VPEs and CTOs at leading SaaS companies. Adam joined me for an interview on Scalying New Heights. Below are a few highlights.\nFirst, a bit of context. As online trading and cryptocurrency explodes, Robinhood has enjoyed a huge increase in new user signups plus expansion of trading on the platform, and this impacted every part of the business and the underlying infrastructure. As new users grew quickly, along with trading volumes, the engineering team needed to be sure they were hitting the new peaks during their own load tests, not as part of the business runtime.\u00a0 Adam explains it this way:\nScaling is about building teams, tools and processes that can continually handle new highs.\nIf you've ever tried to build these kinds of tools and these kinds of frameworks, you know it\u2019s a non-trivial engineering task.\u00a0 As with most things in business, leadership is key, and leaders themselves are continually challenged to set appropriate goals, push teams to change or adopt new technology, and find the right balance between innovation and optimization. Adam\u2019s experience is emblematic of what happens when the business objectives and the tools to achieve them get mixed up.\nWhen Adam joined Robinhood there was a new project to implement \u2013 Kubernetes \u2013 and at first, he thought that sounded like a good idea. \u201cThe business objective was simply to move to Kubernetes,\u201d he says. He had come from Facebook, where they had sophisticated and high-quality frameworks for deploying different kinds of jobs in a very abstract way. You didn\u2019t have to think about it, you could deploy new code and it just worked. The marketing materials for Kubernetes, and the demo, made it look like exactly what they needed, so he jumped on board. But years later, they\u2019re still working on the migration, and they\u2019ve made a lot of progress, but it was more difficult than expected. In this respect, they are not alone.\nWhen you deploy a technology like Kubernetes, it's not just one team that\u2019s impacted, it\u2019s every team \u2013 infrastructure platform, security, product, and so forth. \u201cWhen you frame the goal in terms of the technology, you leave room for interpretation by all of these teams, and each one of these teams will bring their own goals, desires, and fantasies to the project,\u201d Adam says. Without a clear and unified business objective, the Kubernetes project had no unified, defined direction. The real goal was to help product teams deploy software quickly. THAT was the objective.\n\u201cIn retrospect, it's pretty easy to say that management and leadership \u2013myself \u2013made a mistake in saying that we wanted Kubernetes,\u201d he explains. \u201cBecause Kubernetes is not what I want. I actually have no real opinion about the technology that we use \u2013 what I want is faster release velocity.\u201d\nUltimately what I really want is immutable infrastructure.\n\u201cI want to get to the point where it's easier and faster to deploy a new build with the fix, than it is to go back and try to change anything.\u201d\nInvesting in scalability\nWhen setting the direction for another major investment in scalability, Adam wanted to avoid the mistakes he felt were made with Kubernetes. A financial services company like Robinhood will have a variety of offerings for customers, such as a crypto service, an equities trading service, an options trading service, etc. The classic way is to scale these things independently. So the options trading service would be sharded against multiple databases, and the crypto service would separately be sharded against multiple databases.\nBut most things at Robinhood are account-based, so this scaling strategy does not work and. Instead, the company has a scaling strategy that takes advantage of this unique aspect of Robinhood's service (e.g., account isolation).\u00a0 A Robinhood financial account is, by design, something that's only accessed by one or maybe a couple of users, so it's desirable to put one account in one shard. This allows separate crypto options and equity services, with a controller that knows the overall sharding is for all the accounts, while the individual services are blissfully unaware, which limits the number of interservice connections, and the number of hosts that need to talk to each other.\nWhat we found, experiencing this hyper growth, is that the second-order effects get you.\n\u201cYou can almost always find a way to scale something by adding more machines, or tuning a database query or something like that,\u201d Adam says.\u00a0 \u201cBut ultimately they may need to access the service location system, or Kafka, and it's these connections that ultimately cause problems.\u201d\nAchievable goals\nRobinhood\u2019s new cell-based architecture increases fault isolation and reduces several of the scaling pain points. The ultimate goal was to achieve a flexible architecture that allowed the company to release reliable code, quickly. Framing the goal in this way allowed measurable milestones, such as achieving fewer connections among hosts, or containing the blast radius when there are production issues. Framing the goals in the right way provides direction for the team and frames every conversation within and among teams, and creates alignment in a more natural way.\n\u201cWe need to set goals that truly advance the cause of the business, not of engineering itself. I often get caught up in the technical details of or the coolness of a new solution. It's really important for leadership to zoom out and ask, what are we trying to do? And maybe even more importantly, how will we know if we're doing it? And when we set goals in these terms, they have a very different flavor and tenor, and they leave a different kind of room for the engineering team to go pursue them.\u201d\nMany of us have spent our whole careers in the tech industry because we love tech. It\u2019s easy to get excited by the next new technology. But as leaders, it\u2019s important we think about the business goals first, and then the best ways to achieve them.\nYou may want to use your own resources, build everything from scratch and customize the heck out of it, and then maintain and develop it yourself. Or you may want to take something off the shelf and just move fast, focusing your own resources on things that are central to the business. This is always the dilemma, and the path through it begins with framing up the results you expect, versus the path to get there. Says Adam:\nYou know, this is a lesson I've learned again and again.