CIO Upfront: ?The main challenge is not about having too few data scientists

No caption

Have some sympathy for the manager of the corporate analytics department. Everyone, it seems, is talking Big Data this and Big Data that. We hear there is a worldwide shortage of data scientists.

Analytics is the new oil. There's not enough of it and we all need more. This should be good news and for many managers it is because it means bigger budgets and more people.

My view is that such enthusiasm is misplaced. The strategic problem is not too few data scientists, poor data quality or keeping track of the spread of open source analytics. These things can all be solved using existing management techniques. The problem is not technical, technological or financial. The problem is historical. The problem is functional. The problem is managerial. These things require new techniques and ways of working.

The corporate analytics manager runs a business unit built on the functional principle of organisation. Henri Fayol pioneered functional organisation in the 1870s, in the era when work was predominantly physical. The analytics business unit is also likely to be part of a decentralised enterprise structure, which Alfred Sloan developed in the 1920s.

The functional model started breaking down in the 1960s and is hopelessly out-of-date in the 2010s. The decentralised enterprise structure began its decline in the 1970's with the growth of multi-product service businesses. The manager of the internal analytics department is working with a 140-year old model originally designed for the mining industry and in a parent organisation derived from the automobile industry.

This wouldn’t be a problem if internal demand for analytics services was predictable and regular. Even then it wouldn’t be too great a problem if quality enterprise analytics wasn’t a minimum condition for the success of digital transformation programs of large organisations over the coming decades.

But analytics demand is unpredictable and irregular, and large organisations that don't get their enterprise analytics sorted out won't be large for long.

Why can’t we all just get along?

Page Break

Complicating the challenge of supplying quality enterprise analytics is the predominant model of managerial relations. Internal competition and external volatility puts pressures on budgets and careers. People don’t always behave nicely. The arguing over ‘ownership’ of analytics between different seats in the C-suite makes a hard job harder.

In a May 2013 article for CIO Should the CIO or CMO Take the Lead on Big Data, Dion Hinchliffe wrote about the need to sort out the culture of parochial managerialism:

One could make the case that in today's increasingly commoditized and outsourced IT world, big data is a chance for a CIO to finally focus on information as the most important currency of the realm, to rise above the less-strategic task of infrastructure management, and finally drive the business.

However, today's CMOs are increasingly becoming the orchestrators of market-connected business functions and building their own IT capabilities to achieve their objectives. In fact, more and more companies are adding a "CIO of marketing" title, and Gartner has predicted that the CMO's technology budget will eclipse the CIO's in a few short years.

Or instead of focusing on their differences and fighting for the same budgets, the CIO and CMO can come together at this critical juncture in the evolution of their business. They can realize that they have the resources to genuinely support each other, that they each have core competencies that the other can learn from, and that they have overlapping yet reinforcing responsibilities for improving how the business works.

[Related: The data science activist]

Sustaining an analytics ecosystem

The internal demand for corporate analytics is outstripping supply. This is one of the key themes explored at length by Ovum analysts Tom Pringle and Surya Mukherjee in their Oct 2015 report ‘2016 Trends to Watch: Analytics’.

Business users are building better analytics skills (which is a good thing) and their growth in learning is overloading the capacity of the functional model. Pringle and Mukherjee note that speed is one of the drivers influencing the increase in demand:

The business environment dictates faster time to value and a two-year deployment cycle will no longer be supported or even contemplated.

Overloaded internal supply and unrelenting pressure from the external environment is pushing business leaders to turn to external vendors providing analytics-as-a-service options that just don’t exist internally. This places pressure on the ability of the CIO to manage integration and duplication risk, which contributes to significant organisational drag. When the response by corporate analytics departments is outlawing external vendors it is the enterprise that suffers.

SAS Users of New Zealand chairman Rohan Light

Large organisations that don't get their enterprise analytics sorted out won't be large for long.Rohan Light

Pringle and Mukherjee go into some detail on this point:

Bringing IT and business together is critical in the modern analytic world as it is not possible to sustain an analytic ecosystem without the participation of both sides. Many enterprises are tasking cloud and services vendors to manage routine deployment issues, but very few yet buy into the outcome-based pricing model. The ultimate responsibility and genesis for ensuring that the organisation benefits from analytics still lies with the business.

The benefits case for analytics should reside with ‘the business’. But what is the business if not actually the ‘enterprise’? The modern organisation runs on information and because the functional structure no longer suffices, we find ourselves looking for a model that reflects the critical requirements of turning data into insight.

This insight can only emerge when managers emphasise the enterprise over the department and the strategic over the functional. A compounding imperative for doing so comes from the increasingly large data sets being pumped into organisations. There is simply no way it can all be leveraged in a functional model. It needs to be distributed across the organisation.

[Related: Data scientist: Most 'in demand' job of the century?]

Exploratory analytics and machine learning

This means opening up tools and platforms for the entire enterprise community. In practical terms this means providing exploratory analytics tools to complement the curated analytics tools deployed by the analytics departments. Pringle and Mukherjee:

Exploratory analytics is ideal for nontechnical users as most of the query is visually driven. It is different from traditional analytics, where the data sets, schema, and queries are already predetermined; a large part of the process is identifying the right data set(s) to use. This has huge implications on how data is sourced, wrangled, governed, or stored.

Exploratory analytics platforms such as SAS Visual Analytics are excellent platforms for business users to learn the fundamentals of analytics, help shoulder the enterprise analytics burden and contribute to generating insights to fuel business growth.

Investment in machine learning will continue apace, but without a balanced investment in exploratory analytics the organisation persists in overloading its functional and managerial structures. The extension of the machine learning base within organisation's is one of the points covered by Polly Mitchell-Guthrie in her December 2015 post on her SAS blog:

Machine learning dates back to at least 1950 but until recently has been the domain of elites and subject to “winters” of inattention. I predict that it is here to stay, because large enterprises are embracing it. In addition to researchers and digital natives, these days established companies are asking how to move machine learning into production.

In terms of basic organisation dynamics, where one part of an enterprise is heavily invested at the expense of another, we create a chain-link problem. This is where the link in the information chain will fail at its weakest point. Overinvesting in one part of the chain (i.e machine learning) doesn’t make the chain stronger.

Communication and communities

Work in the modern era is about knowledge. Knowledge emerges from the interplay between information and communication. Communication occurs between people. People form communities. Where one community is privileged over another communication doesn’t improve. Where information is held back or released in bits and pieces, the capacity for the organisation to learn is constrained.

Sandy Pentland goes into detail about the importance of communities and analytics in his 2014 book ‘Social Physics’:

If people are to work together efficiently, there needs to be what is called network constraint: repeated interactions between all of the members of the group – not just between a leader and the members, or between the members and the entire group (as at a group meeting). The extent to which good network constraint has been achieved can be tested by asking if the people you talk to also talk to each other.

This emphasis on encouraging communication helps realise enterprise analytics strategy. By taking a community approach to enterprise analytics, senior leadership can leverage network effects and release the growth of data driven insight from the bottleneck of the functional model. By distributing responsibility for analytics across the enterprise, senior leaders will encourage learning and recombination, two critical elements of establishing a data driven culture.

Doing so will lead to complications in procurement, governance and budget allocation. But these can all be managed using existing techniques, as discussed by Eran Levy in his December 2015 blog post on Analytics Bridge, Simple Analytics is Good for Business. On the point of extending access across the enterprise, Eran wrote:

As the research demonstrates, top performing companies have found ways to enable broad access to data for users across business departments, and as a result report increased analytical usage of data as well as higher rates of user satisfaction with the BI tools in place.

By distributing responsibility for analytics across the enterprise, senior leaders will encourage learning and recombination, two critical elements of establishing a data driven culture.


In my capacity as chairman of the SAS Users of New Zealand I’ve taken the orienting theme that New Zealand needs a world class analytics capability. Over the last two years speaking with a wide cross-section of analytics users and consumers, I've concluded that the best way to build such a capability is to broaden the base of the wider analytics community.

This community is far deeper and broader than any functional organisation can effectively represent or leverage. In our 2016 annual event we have chosen the theme 'how are you using analytics to change perceptions?' For me this means about changing the perception of what an analytics professional looks like. My view is that by stepping back and taking a broader look at the industry we will be able to truly leverage the transformational potential of analytics.

The enterprise analytics challenge is one that, if neglected or left unsolved, will significantly degrade an organisation’s opportunities to turn the corner and become data driven and insight led. Taking a community approach is the smartest way to meet that challenge.

Rohan Light ( and @rl_rohan) is director of Decisv, a management business that provides risk-based decision-focused foundation management services. He co-founded the Enterprise Analytics Forum, a community of practice that meets to discuss issues relating to the fundamental challenges analytics to pre-digital business models. He teaches strategic thinking as an Associate for Victoria University of Wellington's Professional and Executive Development. He is Chairman of the SAS Users of New Zealand and most recently worked for Inland Revenue's Change Group as Portfolio Risk Specialist.

Send news tips and comments to

Follow Divina Paredes on Twitter: @divinap

Click here to read digital editions of CIO New Zealand

Sign up for CIO newsletters for regular updates on CIO news, views and events.

Join us on Facebook.

Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, CDOs, COOs, CTOs and senior IT managers.

Copyright © 2016 IDG Communications, Inc.

7 secrets of successful remote IT teams