The dirty secret of data analytics: Culture of honest inquiry required

In the beginning there should have been a culture of honest inquiry.

Instead there was the internal rate of return (IRR) — a polynomial formula for computing return on investment that only trained accountants could master. IRR defined business value in a single dimension of analysis: cash flows. The gods of accounting looked on it with favor, and so it was, theologically speaking, good.

Then Dan Bricklin invented the electronic spreadsheet. Like Prometheus bringing fire to we mere mortals, Bricklin and his spreadsheet let all humanity calculate IRRs for ourselves — and once anyone could compute IRRs for themselves, they did.

The problem? Managers quickly figured out how to do more than compute it. They learned to adjust their assumptions — poking, prodding, and tweaking parameters — until their IRR generators infallibly arrived at the answer they wanted.

The result? Cultures across corporate America in which data lakes, data marts, data warehouses, and analytics software have become little more than platforms for managerial parameter tweaking. They’re used, not for illumination, but as ammunition by those who start with the decision they want and work backward to find the filters and parameters needed to support it. And this is the dirty secret that is sinking your analytics strategy.

From the perspective of organizational dynamics, “culture” is defined as “how we do things around here.” It’s the learned behavior people exhibit in response to their environment, which, in the business world, mostly consists of the behavior of those they work with, and most especially, the behavior of the leaders they deal with every day.

Culture is a self-reinforcing feedback loop. Changing it, to a culture of honest inquiry or any other cultural characteristic, means re-engineering the loop. It requires patience, persistence, and self-awareness.

Want your organization’s analytics strategy to really work? You’ll need to evolve from the usual management game-playing to a culture of honest inquiry. Here are some tips and techniques to get you started.

1. Exhibit the curiosity gene

If you don’t want to know What’s Really Going On Out There, nobody who reports to you is likely to care either. Your dashboards, financial reports, and other forms of organizational listening are there to make you smarter and better informed. If that isn’t what you want, don’t bother building them.

2. Confidence comes from doubt

Certainty, in contrast, comes from arrogance — usually, lazy arrogance. If someone is confident and can explain why, wonderful. If their certainty pre-empts everyone else’s ability to make their case, that person is a bump on the pickle of progress. Ignore them.

3. Start every decision by deciding on the decision process

You don’t have to be in charge to encourage this habit. Just ask the question, “How are we going to make this decision?”

Start with the decision and you start with an argument — an interaction in which every participant’s goal is to win, or, failing that, to refuse to lose. Starting with a conversation about how to go about making the decision changes the discussion, to how to create confidence in the outcome. The results: A better decision, a stronger consensus, and a few more employees who see the benefit of honest inquiry.

4. Don’t create disincentives for honesty

If you ask for honest data, and use it to “hold people accountable,” you won’t get honest data. Why would you? The superior alternative is to employ people who take responsibility without needing the threat of punishment — what “holding people accountable” generally means — to motivate them. This works much better. And as a pleasant fringe benefit, it calls for less effort on your part.

5. The ‘view from 50,000 feet’ is for illustration, not persuasion

A high-level strategic view is essential for focusing the efforts of the organization. “High-level logic,” in contrast, is oxymoronic. Detailed evidence and analysis is what determines whether the high-level view makes sense, or just looks good in the PowerPoint.

6. Beware of anecdotes and metaphors

They’re also useful for illustration, and also shouldn’t be used for persuasion.

Metaphors are an outstanding tool for explaining your point of view; anecdotes can be handy if you need to demonstrate that something is possible. For anything else you need statistically valid evidence.

Yes, Disraeli said there are three types of lie. He miscounted. Argument by anecdote is far more pernicious than argument by statistics, and argument by metaphor is even worse. And yes, you do have to understand statistics well enough to evaluate the evidence. That’s part of your toolkit.

7. Evidence too far removed from the original source is suspect

Don’t trust summaries of summaries of summaries, especially if they tell you what you want to hear. Even with the best of intentions the game of telephone is in play.

8. Don’t trust your gut

By all means listen to it. Your “gut” is the voice of your experience which isn’t something to ignore. But don’t trust it, because your experience is both too biased and too small to constitute a decent statistical sample.

9. Build your culture of honest inquiry one decision at a time

Preaching and lecturing don’t work, and they work even worse when the proselytizers don’t practice what they preach. So just remember: the best way to build a culture of honest inquiry is one decision at a time.

And for all forms of culture change, the most important tool at your disposal is a mirror. Look in it whenever How We Do Things Around Here looks like it’s headed in the wrong direction. Ask yourself what you might be doing to pollute the feedback loop.

Even more important, figure out how to keep yourself from making the same mistake again. We all make mistakes. Make the same mistake over and over again, though and it isn’t a mistake anymore.

It’s a bad habit, one nobody sells a patch or new platform to help you break.