DataOps: Building the Data-to-Insights Pipeline

A new IDC InfoBrief draws an interesting analogy between water pipelines and data pipelines. Water comes from different sources and must pass through various stages of cleaning, processing, and analysis before it is delivered to home owners’ taps. Likewise, IDC notes, business data also comes from multiple sources and progresses through various stages before it delivers maximum business value.

In the brief, IDC identifies four core stages in a “Data-to-Insights” pipeline: Identify Data, Gather Data, Transform Data, and Analyze Data. Collectively, these stages also constitute the core elements of an emerging discipline called DataOps. Much as DevOps has accelerated and systematized software development, DataOps aims to accelerate the collection, processing, and analysis of fresh data.

Central to Qlik’s DataOps model is the concept of “Active Intelligence.” Rather than basing business decisions on data that has been preconfigured and placed in curated data sets, Active Intelligence continuously draws upon relevant and current data in real time.

In order to provide the agility and flexibility that Active Intelligence requires, DataOps processes must leverage a collection of technologies and best practices. In essence, the fundamental challenge of DataOps is to create a unified, and massive, data supply chain that is supported by automation and, increasingly, by artificial intelligence (AI).

AI comes into play by helping organizations identify which data is most relevant and actionable in relation to any given need. And, rather than locking business users into preset questions and searches, AI can help support a more free-form and interactive approach to data query and analysis. Accomplishing this goal requires tools that can perform on-the-fly calculation to identify and deliver the most meaningful data from a wide range of different data sources.

The need for such sophisticated data sourcing, processing and management is clear, based on the results of a global survey including 1,200 organizations conducted by IDC. As reported in the IDC InfoBrief, during the past 12-18 months:

• 45% of the organizations began using new internal data
• 40% began using new external data
• 45% began using new data types
• 47% began using new data analytics

In addition, 38% of the surveyed organizations said they have also begun tracking new key performance indicators (KPIs) during the past 12-18 months. As IDC notes, the introduction of new KPIs means the organizations are asking new questions, and may require new data sources to answer them.

As part of its work in this area, IDC has developed an online survey that enables organizations to quickly determine their Data-to-Insights (D2I) score in relationship to their peers. A customized plan of action to increase that score is also provided.

Having a well-integrated and capable data-to-insights pipeline isn’t merely an abstract concept. As we’ll discuss in an upcoming post, IDC finds that organizations with a strong pipeline perform better than their peers in both business decision-making and, consequently, in business outcomes.

Click here to learn more about how Qlik can help your organization implement and benefit from a powerful DataOps, Data-to-Insights capability.