Big Data, Little Brain

Lured by the promise of AI and machine learning, and by the ease of collecting and storing data about every facet of every activity in our daily lives, leaders across most industry sectors have embraced big data as the next “big thing” in helping them manage a variety of corporate functions, including marketing, pricing, supply chain, operations, finance and more.

I am a big fan of quantitative approaches, I have worked in and around data science for nearly three decades, and I am a firm believer in the adage that you can’t control what you can’t measure. From this perspective, I have been delighted to see a growing trend in the past decade for leaders across all industries and corporate functions to embrace quantitative approaches that help them make better decisions.

However, I believe that the adoption of big data may have gone past the point of diminishing marginal returns, and is even causing problems in some areas.

My greatest concern is what I call the “Big Data, Little Brain” phenomenon: leaders who abdicate their knowledge and rely excessively on data to guide their decisions. Specifically, in a typical big data project, a leader might engage an external or internal team to collect and process data, hoping to extract insights related to a particular business problem. The big data team has the expertise needed to wrangle raw data into usable form, and to select algorithms that can identify patterns and extract information. The results of this exercise are then presented to the leader through reports, charts, infographics and other types of visualization.

The problem is that, in this typical scenario, the leader’s role is limited to using her expertise to make sense of the information prepared by the big data team, and to use that information to guide decisions; her knowledge of the business has no bearing on how the data is processed, or what algorithms are used to extract information from the data. This is because most business leaders are not experts in data science, while most data scientists are not an experts in the business areas of the leaders they support.

It is important for decision-makers to understand the limitations of big data approaches, and to explore methodologies that let them incorporate their knowledge and expertise into the entire decision-making process, not just in the interpretation of results provided by data scientists. This includes more process-oriented, qualitative methodologies like Design Thinking, as well as more quantitative methodologies such as Agent-Based Simulation (ABS). ABS in particular is gaining popularity in a number of other fields, including economics, epidemiology, social science, medicine, finance, transportation, tourism and many more.

For the past two decades I have applied ABS to a variety of problem domains, from managing personnel for the U.S. Navy, to improving the drug development pipeline for a major pharmaceutical company, to increasing energy efficiency in buildings. In 2010, I co-founded Concentric, a company that uses ABS for marketing analytics. Back in 2014, Concentric was retained by a leading automaker to help it plan the launch of a new model. Concentric recommended doing the launch six months earlier than the client was planning. In 2016, the automaker launched the model as recommended; a year later, it found that Concentric’s simulation had predicted monthly sales for the first year with 93% accuracy.

More recently, advertising agency MediaStruction used the Concentric platform to help a large commercial bank forecast sales of its consumer checking, business checking and home equity products. At the close of Q1, it ran a blind comparison and found that the model’s predictions for these three product categories deviated from actual sales only by 1.1%, -0.4% and -4.1%, respectively.

In summary, by embedding the manager’s expertise into a predictive model, ABS is able to solve complex problems in a transparent way with a high degree of predictive accuracy. The increased availability of commercial ABS tools and didactic materials suggest that this “Big Brain, Big Data” approach is poised to revolutionize business management.