Making IT Work: The Metrics Trap

As an ambitious young computer science and economics major who desperately wanted to be a writer, I got very good at being taken out for lunch by editors. I did my homework, I studied the magazines, and during the meal I never failed to ask my host, “So, what’s the biggest problem you have with your freelance writers?” (The idea was to assure them I would never repeat such a mistake.) Without exception, the editor would lean back, sigh, shake his head and say, “The biggest problem we have with our writers is that they write too long. We always end up having to make cuts.”

Note to self: Don’t write long!

Coffee and dessert finally arrive. The conversation has gone well. Now comes the moment where I ask The Big Question: “So, umm, how much do you actually pay?” Without exception, my editor would lean forward, nod and say, “Oh, about a dollar a word.”

Duh. Dollar-a-word/Writes-too-long. Tomayto/Tomahto. I was shocked by how these editors were oblivious to the obvious connection. But why pick on editors? I found the same perverse dynamic when I was chatting with a group of CIOs at an MIT dinner.

In recent years, how have most IT organizations measured?and rewarded?programmer productivity? LOCs?lines of code?per unit time. And then we’re surprised that we end up with bloated, inefficient and poorly documented systems? Who are we kidding?

Perverse metrics linked to perverse incentives always lead to perverse results. That is an iron law of human nature and economics. Want to know the secret to successful implementations? Define what a “successful implementation” is and have the courage to ask: Is this a definition that creates perverse incentives? Or is this a definition that leads to metrics we can meaningfully recognize and reward?

Today’s IT is awash in metrics that misalign expectations, processes, rewards and outcomes. Too many CIOs spend too much time defining what IT and the business are trying to do and not nearly enough time on the metrics and incentives that will make it easier and more economical for their people to do it. Perverse, unhappy outcomes are the consistent result.

The obvious examples come from cost-cutting and outsourcing. Many implementation metrics are biased in favor of IT costs saved?by reducing calls to the help desk, for example?rather than measuring improvements to a business process. These metrics reward IT managers who “cut costs” and “work to put themselves out of a job.” In this regard, the incentives work perfectly: IT budgets are indeed cut, and those IT managers get a lovely severance package.

Of course, as every reader of this magazine well knows, many of those costs stripped out of IT ultimately reappear elsewhere in the budgets of the business units: Servers and databases don’t run themselves. Yes, the IT managers are gone, but it’s often unclear for more than a year who exactly is going to be held managerially accountable for the systems left behind.

In other words, the IT budget is down dramatically?but are the actual costs to the enterprise really lower? IT headcount is undeniably slashed, but are the managerial costs of IT coordination, modification, maintenance and procurement really that much cheaper?

The simple and honest answer to those questions for most organizations is: We haven’t a clue. Why not? Because, of course, that’s not what these companies have been measuring. They’ve been measuring IT expenditures and investments with the same level of rigor and sophistication that they once measured programmer productivity. Why are they doing that? For the most obvious reason: They’re not being rewarded for cutting costs for the organization; they’re being rewarded for cutting IT costs. Anybody who believes that cutting IT budgets and reducing the enterprise IT spend are synonymous should not even think of being a CIO.

The dirty little secret here is that some of today’s most “successful” CIOs are executives who’ve dramatically cut their own budgets while effectively raising the IT spend for the rest of the enterprise. True, some of this spend may be better aligned with individual business unit aspirations. But then, that hardly reflects any strategic or managerial prowess by the CIO. Remember: Perverse metrics linked to perverse incentives always yield perverse results.

At a series of workshops at Microsoft last year, it was striking to hear enterprise architects discuss the gap between metrics that measure improvements in technical capacity versus those that measure enhancements in business capability. Enterprise architects have conflicted loyalties. Most IT organizations have their investments and incentives more clearly aligned with technical capacity improvements?which are far easier to measure?than with improved business capability?which requires P&L managers to crisply define what they want IT to accomplish. The business roles and goals of architects are in unavoidable conflict.

So the workshop participants played with the notion of how we should measure ROTCI?return on technical capacity improvements?versus ROBCI?return on business capability improvements. Should ROBCI drive ROTCI? Or should ROTCI be used to encourage line managers to push for more ambitious ROBCI? How could these measures be better aligned? The group ended up using metrics as a medium?not just a technique?for thinking through and articulating what capacity and capability should mean throughout the enterprise.

It’s dangerous for CIOs to take the approach that ROTCI always drives ROBCI. Instead, they need to better understand how line managers define business capability improvements and invest in technological capacity accordingly. And once they develop these new metrics, they need to come up with better incentives to achieve the business goals.

The era of “dollar-a-word/lines-of-code” metrics and remuneration simply doesn’t cut it. Quite frankly, those kinds of lazy measures and incentives never did, but organizations could once get away with that kind of laziness. Today, they can’t. They shouldn’t. Smart CIOs should be in the vanguard of innovating a new generation of IT incentives. They’ve got every incentive to do so.