One of the big concerns I have about analytics is that it’s used badly more often than not. When I say badly I mean analytics gets used to confirm a bad decision that was already made rather than using it to assure the right decision gets made.
You see this most obviously in politics and contentious issues where the desire to prevail exceeds significantly the desire to do the right thing. One of the most painful instances to watch was in the last presidential election. Democrats and Republicans used analytics heavily, but the Republican process was so flawed it led them to a false sense of victory and materially contributed to their loss.
Then I was reading a Washington Post story on Gun Control looking at a report from Columbia University, which in turn looked at a whopping 130 studies done over 64 years on gun control and found that a lot of controls (like gun buyback programs) are a waste of money, and those that work (like denying guns to mentally ill people) should be pretty obvious.
A key part to any kind of research has to be a willingness to accept the facts because, if management doesn’t want to accept the facts, the analytics effort is likely to fail and the people responsible for it are at job risk. And, most important, if you work for a company with executives who are actively participating in the falsification of reports, even if they are “just” market analysis, you’ll likely be laid off at some point as the company fails around you.
[ Related: 21 data and analytics trends that will dominate 2016 ]
I’m using gun control as the subject of this because it is topical and the dispute has little to do with the 2nd Amendment or saving lives. It is about one side attempting to control the other, and it means both sides tend to use statistics as a blunt instrument and the statistics are often compromised as a result.
So let’s jump to this massive study.
Not validating a conclusion
On the gun control side, one of the popular practices is to put in place gun buyback programs and then showcase how many guns were taken off the street. Everyone feels good because they accomplished something. But the study concluded that this activity has no significant impact on gun deaths, which means this money, which could have been spent on things that actually made a difference, was wasted.
Years ago I had a meeting with a large software company that told me they were killing a program that allowed employees to take a copy of a popular business application home and use it for free as long as they had a legal license to use it at work. They said the decision was made because IT demanded it. This seemed insane -- it turned out what the company had done was tell their clients that if their employees made illegal copies they’d be sued and the clients said they didn’t want to be sued. So the firm made the leap in thought that the clients didn’t want the work at home program. But what the customers were really saying was simply that they didn’t want to be sued. That distinction could have led to a decision that preserved the program, but limited customer liability.
In the case of gun buybacks, because the payments are a fraction of what a good gun is worth people that intend to use the guns or intend to sell them to others don’t participate. As a result, there is no impact on the actual problem. The program simply eliminated guns that weren’t in use and apparently aren’t a danger. Yet the programs created the false impression it was making progress.
This is why it is important to make sure the results are consistent with the program goals. However, because this may make the decision-maker look bad, often that effort is never made and bad decisions aren’t corrected.
[ Related: 6 analytics trends that will shape business in 2016 ]
Treating requirements like multiple choice questions
In reading through the study one thing is painfully clear -- most of the gun control laws implemented in the U.S. have had very little impact on anything but suicide rates. There are clearly conflicts but the only constant is that restrictions do seem to have a positive impact on folks that would kill themselves.