3 Ways DataOps Reduces Data Complexity

We’ve come to a data crossroads. Between regulations such as the California Consumer Privacy Act (CCPA) and General Data Protection Regulation (GDPR), shifting architectures with multicloud strategies, and new uses cases with artificial intelligence (AI)/machine learning, there’s a need to simplify and automate management of data. As some already know from experience, this is often easier said than done. Many organizations are hobbled by traditional data warehouses or Hadoop data lakes, which can be complex and difficult to manage. But there’s good news, though. Despite everything, organizations that embrace a DataOps approach can get a jump on the competition by improving data management and boosting innovation.

DataOps, which builds on the ideas and practices of DevOps, is a form of enterprise data management. DataOps “applies the principles behind DevOps to the world of data management to create three layers that constitute a new data management infrastructure based on agile data pipelines, governance and instrumentation, and operations agility,” according to Hitachi Vantara.

In its most basic form, DataOps breaks down barriers between data experts and data users; provides a highly sophisticated form of data governance; and introduces new policies related to retention, encryption, and security. Each of these improvements makes data management easier and takes much of its complexity out of the equation. DataOps also overhauls a company’s data organization and the way data is automated and used.

Moving to a DataOps Model

There are three major ways technology will change when organizations move to a DataOps model.

• Increased use of metadata: One of the main bottlenecks in traditional data management is so-called dark data, or data that is collected but not known or understood. When organizations expand their use of metadata, IT and users have a better way to tag, categorize, and search for data. From a governance perspective, this makes it easier to identify sensitive data and implement controls and rules around it. This also helps with automation, as it’s easier to automate what is clearly labeled and categorized.

• Growing use of automation: Automation is key in the age of growing volumes of data. Under the DataOps umbrella, database functionality such as ETL and tasks including data prep, data quality, controlling data pipelines, repositories, and data management infrastructure can be automated. Automation can also create more self-service capabilities for users.

• Expansion of policy-driven control and configuration: Data users need applications and controls that make it easier for them to do their jobs. They need to make sure, for instance, that all data of a certain type is stored in a specific location and that all personally identifiable information is masked. Using policy-driven control and configuration, data users typically get exactly what they need, when they need it. This control also makes it easier to limit or manage access and assign privileges.

To learn more about how DataOps can help your organization, read Hitachi Vantara’s recent whitepaper, Bringing DataOps to Life.