Data is your biggest digital hurdle. DataOps will help you clear it

I spend a lot of time talking to senior executives at large companies, and if there’s one phrase I hear them all talk about consistently — perhaps a little too often —  it’s “Digital Transformation.”

Ask 10 people what it means and you’ll get 10 different answers, but in broad brushstrokes it means using the power of software to transform and improve business processes large and small, and to study the data that software generates for new ways to boost efficiency, reduce costs or make money.

And we can all probably recite the list of things that have made that easier: Cloud computing; agile development and collaboration tools. You can lump them into shorthand phrase “DevOps,” and in general, they have made building software easier and faster.

Yet it’s true that companies can get everything right about DevOps and still fail at digital transformation. And most do fail — about 84 percent by one reckoning. One big reason they do fail is that they ignore the complexities around handling data.

I got to talking about this recently with Eric Schrock, CTO at Delphix, a company founded in 2008 and based in Redwood City, Calif. that is focused on the fundamental problem of making data easier to work with.

Companies have in the last decade or more developed an “unquenchable thirst for data,” he says. And that fact bumps up against a fundamental problem: While every aspect of the world of software has become light, agile, and streamlined, data is nothing like that.

“Data is the new competitive advantage. It doesn’t matter what business you’re in,” Schrock told me in a recent interview. “It doesn’t matter if you’re an analyst or a data scientist or the head of marketing. Everyone needs data.”

Sure everyone needs it, and there’s more of it than ever before. But data is complex, heavy, expensive to maintain, and difficult to move. At a large corporation the scale of managing it can be staggering, and this fact can cause tremendous friction throughout a company’s digital operations.

One reason is size. A typical corporate database may be one to 100 terabytes, and the number of those databases can range from dozens to tens of thousands amounting to hundreds of petabytes worth of raw data.

And while developers, marketers, analysts, and even data scientists are all clamoring to get their hands on live data, they often end up having to settle for one of many bad workarounds: They work with a limited subset of the data they need; They substitute old, stale data in place of the live production data that’s services as the lifeblood of modern applications. 

What companies need, Schrock argues, is a new approach to working with data that like DevOps, breaks down the barriers to progress. He calls it DataOps, which strictly speaking isn’t a new phrase, but one whose meaning he’d like to extend: “The alignment of people, process, and technology to enable the rapid, automated, and secure management of data.” 

The goal he says, is to eliminate the inherent pain points that come with managing data so that everyone who has a need to work with it can do so, and get their jobs done.

No surprise, Delphix is in exactly this business, and it’s backed by $120 million in funding from an impressive list of investors that includes Greylock Partners and In-Q-Tel. In 2015 Fidelity Investments led its $75 million D round that valued Delphix at nearly $1 billion.

The Delphix platform connects to production data wherever it lives. From there it builds a detailed timeline, tracking changes and additions that’s compressed and stored efficiently. The result is an virtual version of the data that’s easy to access and easy to work with for the purpose of building an application, running an analysis or pretty much anything else data might be used for.

Developers no longer need to ask their database administrators for a data refresh. They can refresh the data themselves and typically wait minutes instead of days. QA testers hunting bugs can do so using real data they can share with other teams. “The point is that everyone gets access to the data they need to get their jobs done,” Schrock says.

The platform is currently sold as an on-premise product and for public clouds.

But DataOps is more than simply a platform. Like DevOps before it, it’s as much about people, organizations and how they interact and share information as it is about labor-saving tools. “You’ve got data operators, DBAs, security people, IT people. They’re all responsible for securing and enabling how that data is distributed. And on the flip side you’ve got data consumers, and historically, they’re always butting heads. We want to get them working together.”

Under DataOps, everyone involved in the data lifecycle has a meaningful part to play in the conversation in how its use evolves. “DataOps is about a cultural shift and the solutions that makes it possible,” he says.

Using data effectively is shaping up to be a critical and frankly fundamental capability that companies will simply have to master in the coming years, Schrock says. Getting the app you use to reach your customer done on time can make the difference between companies who win and lose in the marketplace.

And it’s a the sort of problem that touches nearly any business: Finance, health care, manufacturing. Delphix’s customers run the gamut from tech players like Hewlett Packard Enterprise and StubHub to financial giants like JPMorgan Chase and the U.S. space agency NASA.

And looking ahead, data will be crucial to the deployment of artificial intelligence and machine learning systems as they arrive on the scene in the coming years. “Everyone wants to mine their data to find some new insight about the marketplace before their competitors,” Schrock says. “But before an AI system can do anything, you have to feed it your data.”