What is your time to answer?

A little over a decade ago I had the great pleasure of hearing a commencement speech for my son. Eric Lander, the leader of the human genome project, described his journey through science and life. He shared that he did not have a clear direction as a mathematics major at Princeton, and the labyrinth of decisions that followed to get him into genetic biology. He also shared the process of science and its adoption into the culture and economy of the modern-day world. His punchline was that it takes a generation to understand and incorporate scientific discoveries into the economy and culture, as my father had once told me.

There are many things that ring true from this extraordinary scientist, but I no longer agree with his assessment of the cadence of innovation. It is faster—much faster. The cycle of innovation that started in chemical laboratories, thousands of experiments, and hundreds of clinical trials, have now been replaced by millions of natural experiments per day. This is not only true of the elite scientific disciplines, it is true in our everyday social life—as we know. Our every communication, indeed our every footstep, can be tracked and tested for its predictive power.

The fuel for this acceleration has been data—lots of data. From micro-assays to blog posts, the explosion of data has been nothing short of meteoric. The challenge, as always, is turning the raw data into observations in the real world that can indicate reliable patterns and signals that provide insights and predict outcomes.

Today, through the combination of big data economics, data analytics, cataloging, and advanced visualization and machine learning, we are able to build an ecosystem that breaks the generation barrier. Rather than waiting decades for discovery to become accepted theory, we can now create insights in days or weeks and act on them immediately. In other CIO.com posts, I have written about how this acceleration has driven predictive M&A activities, dramatic reductions in data management costs, and addressed the issue of retaining top talent.

Further consolidation of the data management and insight analytics pipeline is underway. The combination of my company, Podium Data, and Qlik, are an example of how the market is structuring itself to provide end to end solutions where data scientists, knowledge workers, and every day consumers can efficiently collaborate to business decisions. Here are several of the principles I think are critical to the future ecosystem:

• Raw to ready: the system should automatically identify dirty data, incorrect data types and semantically ambiguous or questionable data. If data is structurally unsound, it cannot be analyzed for patterns and insights.
• Self-service shopping: information seekers should be able to browse, review and shop for data through a smart catalog that is well-documented and available. The democratization of data and analytics expands the community from elite data scientists to a broad set of consumers with access to well-vetted, governed data. A critical human component of this expansion is data literacy to ensure the workforce can take full advantage of these new capabilities.
• Rapid iterations: analysts should be able to load, access, prepare, and analyze data in minutes without IT professionals in the loop. Unlike the traditional approach of silo’d analytic sandboxes, the new paradigm provides a common platform that manages the data throughout the DataOps life cycle from discovery to production. This further connects the communities of data scientists and business analysts and supports crowd sourcing of information such as the most popular or reliable data sets.

These principles serve to optimize a fundamental analytics metric I defined 10 years ago: What is your time to answer? We know that companies who can deliver answers in hours instead of days (and days instead of months) not only save time and money—they actually transform the business. Analytics start to inform urgent business decisions, processes become instrumented for optimization, data and insights become new products. Just look at how companies with rich data and agile analytics (Amazon, Google) are attacking traditional markets (insurance, banking, retail).

Corporate boards and C-suite executives are launching strategic digital transformation programs to compete in this new world. The lifeblood of these programs is an agile, integrated data and analytics ecosystem that accelerates time-to-answer and enables a rapid test-and-learn cycle.