Building and managing infrastructure yourself gives you more control \u2014 but the effort to keep it all under control can take resources away from innovation in other areas. Matt Doka, CTO of FiveStars, a marketing platform for small businesses, doesn\u2019t like that trade-off and goes out of his way to outsource whatever he can.\n\nIt shows in his reluctance to run his own servers but it\u2019s perhaps most obvious in his attitude to data engineering, where he\u2019s nearing the end of a five-year journey to automate or outsource much of the mundane maintenance work and focus internal resources on data analysis.\n\nFiveStars offers small businesses an online loyalty card service \u2014 the digital equivalent of \u201cbuy nine, get one free\u201d stamp cards \u2014 that they can link to their customers\u2019 telephone numbers and payment cards. Over 10,000 small businesses use its services, and Doka estimates around 70 million Americans have opted into loyalty programs it manages. More recently, it has moved into payment processing, an option adopted by around 20% of its clients, and offers its own PCI-compliant payment terminals.\n\nRecording all those interactions generates a prodigious amount of data, but that\u2019s not the half of it. To one-up the legacy payment processors that just drop off a terminal and leave customers to call for support if it stops working, FiveStars builds telemetry systems into its terminals, which regularly report their connection status, battery level and application performance information.\n\n\u201cThe bulk of our load isn\u2019t even the transactions, the points or the credit cards themselves,\u201d he says. \u201cIt\u2019s the huge amounts of device telemetry data to make sure that when somebody wants to make a payment or earn some points, it\u2019s a best in class experience.\u201d\n\nFiguring that out from the data takes a lot of analysis \u2014 work that the 10-person data team had less time for since just maintaining their data infrastructure was eating it all up.\n\nThe data team that built the first version of FiveStars\u2019 data infrastructure started on the sales and marketing side of the business, not IT. That historical accident meant that while they really knew their way around data, they had little infrastructure management experience, says Doka.\n\nWhen Doka took over the team, he discovered they had written everything by hand: server automation code, database queries, the analyses \u2014 everything. \u201cThey wrote bash scripts!\u201d Doka says. \u201cEven 10 years ago, you had systems that could abstract away bash scripts.\u201d\n\nThe system was brittle, highly manual and based on a lot of tribal knowledge. The net effect was that the data analysts spent most of their time just keeping the system running. \u201cThey struggled to get new data insights developed into analyses,\u201d he says.\n\nBack in 2019, he adds, everyone\u2019s answer to a problem like that was to use Apache Airflow, an open-source platform for managing data engineering workflows written in and controlled with Python. It was originally developed at AirBnB to perform exactly the kinds of things Doka\u2019s team were still doing by hand.\n\nDoka opted for a hosted version of Airflow to replace FiveStars\u2019 resource-intensive homebrew system. \u201cI wanted to get us out of the business of hosting our own infrastructure because these are data analysts or even data engineers, not experienced SREs,\u201d he says. \u201cIt\u2019s not a good use of our time either.\u201d\n\nAdopting Airflow meant Doka could stop worrying about other things besides servers. \u201cThere was a huge improvement in standardization and the basics of running things,\u201d he says. \u201cYou just inherit all these best practices that we were inventing or reinventing ourselves.\u201d\n\nBut, he laments, \u201cHow you actually work in Airflow is entirely up to the development team, so you still spend a lot of mind cycles on just structuring every new project.\u201d And a particular gripe of his was that you have to build your own documentation best practices.\n\nSo barely a year after beginning the migration to Airflow, Doka found himself looking for something better to help him automate more of his data engineering processes and standardize away some of the less business-critical decisions that took up so much time.\n\nHe cast his net wide, but many of the tools he found only addressed part of the problem.\n\n\u201cDBT just focused on how to change the data within a single Snowflake instance, for example,\u201d he says. \u201cIt does a really good job of that, but how do you get data into Snowflake from all your sources?\u201d For that, he adds, \u201cthere were some platforms that could abstract away all the data movement in a standardized way, like Fivetran, but they didn\u2019t really give you a language to process.\u201d\n\nAfter checking out several other options, Doka eventually settled on Ascend.io. \u201cI loved the fact there was a standard way to write a SQL query or Python code, and it generates a lineage and a topology,\u201d he says. \u201cThe system can automatically know where all the data came from; how it made its way to this final analysis.\u201d\n\nThis not only abstracts away the challenge of running servers, but also of deciding how you do work, he says.\n\n\u201cThis saves a ton of mental load for data engineers and data analysts,\u201d he says. \u201cThey\u2019re able to focus entirely on the question they\u2019re trying to answer and the analysis they\u2019re trying to do.\u201d\n\nNot only is it easier for analysts to focus on their own work, it\u2019s also easier for them to follow one another\u2019s, he adds.\n\n\u201cThere\u2019s all this documentation that was just built in by design where, without thinking about it, each analyst left a clear trail of crumbs as to how they got to where they are,\u201d he says. \u201cSo if new people join the project, it\u2019s easier to see what\u2019s going on.\u201d\n\nAscend uses another Apache project, Spark, as its analytics engine, and it has its own Python API, PySpark.\n\nMigrating the first few core use cases from Airflow took less than a month. \u201cIt took an hour to turn on, and two minutes to hook up Postgres and some of our data sources,\u201d Doka says. \u201cThat was very fast.\u201d\n\nReplicating some of the workflows was as easy as copying the underlying SQL from Airflow to Ascend. \u201cOnce we had it working at parity, we would just turn the [old] flow off and put the [new] output connector where it needed to go,\u201d he says.\n\nThe most helpful thing about Ascend was it would run code changes so quickly so the team could develop and fix things in real time. \u201cThe system can be aware of where pieces in the workflow have changed or not, and it doesn\u2019t rerun everything if nothing\u2019s changed, so you\u2019re not wasting compute,\u201d he says. \u201cThat was a really nice speed up.\u201d\n\nSome things still involved an overnight wait, though. \u201cThere\u2019s an upstream service you can only download from between 2 a.m. and 5 a.m., so getting that code just right, to make sure it was downloading at the right time of day, was a pain but it wasn\u2019t necessarily Ascend\u2019s fault,\u201d he says.\n\nMobilizing a culture shift\n\nThe move to Ascend didn\u2019t lead to any major retraining or hiring needs either. \u201cBuilding is pretty much zero now that we have everything abstracted,\u201d Doka says, and there are now three people running jobs on top of the new systems, and around six analysts doing reporting and generating insights from the data.\n\n\u201cMost of the infrastructure work is gone,\u201d he adds. \u201cThere\u2019s still some ETL work, the transforming and cleansing that never goes away, but now it\u2019s done in a standardized way. One thing that took time to digest, though, was that shift from what I call vanilla Python used with Airflow to Spark Python. It feels different than just writing procedural code.\u201d It\u2019s not esoteric knowledge, just something the FiveStars team hadn\u2019t used before and needed to familiarize themselves with.\n\nA recurring theme in Doka\u2019s data engineering journey has been looking for new things he can stop building and buy instead.\n\n\u201cWhen you build, own, and run a piece of infrastructure in house, you have a greater level of control and knowledge,\u201d he says. \u201cBut often you sacrifice a ton of time for it, and in many cases don\u2019t have the best expertise to develop it.\u201d\n\nConvincing his colleagues of the advantages of doing less wasn\u2019t easy. \u201cI struggled with the team in both eras,\u201d he says. \u201cThat\u2019s always part of a transition to any more abstracted system.\u201d\n\nDoka says he\u2019s worked with several startups as an investor or an advisor, and always tells technically minded founders to avoid running infrastructure themselves and pick a best-in-class vendor to host things for them \u2014 and not just because it saves time. \u201cYou\u2019re also going to learn best practices much better working with them,\u201d he says. He offers enterprise IT leaders the same advice when dealing with internal teams. \u201cThe most consistent thing I\u2019ve seen across 11 years as a CTO is that gravity just pulls people to \u2018build it here' for some reason,\u201d he says. \u201cI never understood it.\u201d It\u2019s something that has to be continually resisted or wind up wasting time maintaining things that aren\u2019t part of the core business.