Lies, damned lies and benchmarks

Volkswagen cheated on the emissions tests for 11 million vehicles. But it’s not the only company that has built lying into its testing and benchmarking.

vw golf tdi clean diesel was 2010 8983
Credit: Mariordo Mario Roberto Duran, CC BY-SA 3.0, via Wikimedia

It always struck me, as I recoiled from the stink of diesel fumes on the interstate, that the notion of “clean” diesel had to be wrong. It turns out I was right — at least in the case of Volkswagen, which cheated on emissions tests for about 11 million vehicles. Now countries are banning sales of the makes that had been programmed to lie about pollution results.

Shocking? Only if you think that VW is the only company that’s ever shipped products that faked test results.

If you haven’t heard about it, several models of diesel VW vehicles built between 2009 and 2015 and diesel Audi A3 models from 2010 to 2015 had software installed in their electronic control modules that sensed when an emissions test was being conducted.

When the car figured out that it was being tested, which it seemed able to do almost perfectly, it would switch to “clean mode.” According to the Environmental Protection Agency, “This results in cars that meet emissions standards in the laboratory or testing station, but during normal operation, emit nitrogen oxides, or NOx, at up to 40 times the standard.”

Able to tout great numbers on the test stand with no performance hit on the road, VW was able to post excellent financial numbers — until it was caught. Now VW CEO Martin Winterkorn has resigned because of the cheating scandal.

Oh, well, so much for VW’s next quarterly results.

Why did the third-biggest carmaker in the world do it? Because cheating was easier than building competitive “clean” diesel vehicles.

When the U.S. adopted stricter pollution rules in 2008 (with the EU following in 2015), VW was caught between a rock and a hard place. It found it impossible to build clean diesels that had good fuel mileage. So it programmed its cars to fib.

I am not among those who find this sordid tale surprising.

I’ve been running and building benchmarks for close to three decades now, and I’ve been an expert witness in cases involving them. And I can tell you that tech companies have been double-dealing on tests since day one.

Back in the 1980s, I and other technology journalists discovered a brand of Ethernet card that knocked the others out of the race. How could it be so much faster than the others? It cheated.

When this card “saw” a then common traffic pattern involving Novell network interface cards (NIC), as my colleague Wayne Rash observed, “it simply passed packets through without processing.

There were others. Many others.

Just a few years after the NIC cheat, for example, Rash encountered a brand of Gigabit Ethernet switches that had results that were too good to be believed. In this case, when tested with real-world data, the switch not only had poor performance, it “failed spectacularly,” as Rash put it.

Another writer and programming buddy of mine, Stephen Satchell, reminded me of another case. This time it was a software trick very reminiscent of what VW did. As he recounted this incident in a recent email, a “C compiler (names removed to protect the guilty) would detect that the Sieve of Eratosthenes [a simple algorithm for finding prime numbers often used to test the speed of compilers] was being compiled, and would inject handcrafted intermediate code that had been highly optimized.” This made the compiler look faster, although the trick could sometimes cause programs to break.

In the ’90s, I also encountered a brand of graphic cards that always won our benchmarks. They were good, but that good? We didn’t think so.

So we looked closer and found that the cards were set to detect our benchmarks. When they did, they either speeded up the GPU or output the correct graphics from embedded memory, depending on the particular cheat.

Like VW, all these companies took shortcuts to make more money on the cheap. People willing to break rules to look better than the competition in “objective” benchmarks and tests will always be with us.

So whenever you see benchmark results that appear too good to be true, go with your gut reaction. It’s all too likely the vendor has been playing games with numbers just so it can swindle you out of some money.

This story, "Lies, damned lies and benchmarks" was originally published by Computerworld.

Download the CIO October 2016 Digital Magazine
View Comments
Join the discussion
Be the first to comment on this article. Our Commenting Policies