How to Measure Real Outsourcing Success (Hint: It’s Not the SLAs)

“There’s a lot of pain around outsourcing deals,” admits Dr. Paul Roehrig, a senior analyst with Forrester. The situation in the $120 billion-a-year IT services market isn’t quite as dire as some would have you believe, says Roehrig. About 57 percent of IT buyers are somewhat satisfied with their primary outsourcer and 22 percent are very satisfied, according to Forrester’s research. But that leaves more than 20 percent indicating it could be better.

“It’s not as bad as a lot of people make it out to be. You’re not inviting disaster into your firm by outsourcing,” Roehrig says. “But there’s a tremendous amount of room for improvement.”

That’s why Roehrig advocates regular checkups on outsourcing deal health. And that doesn’t just mean looking at performance against service-level agreements (SLAs) or price benchmarking. “All the service providers are capable of delivering a basic suite of services,” says Roehrig. Vendors tend to come within plus or minus 2 percent on most SLAs, so they’re “not the real driver of satisfaction” or dissatisfaction, he says. When it comes to cost savings, most IT buyers are able to save an average of 12 percent to 17 percent by outsourcing, according to outsourcing adviser TPI, but there’s lots of variation based on what you’re outsourcing and where. Buyers can try to use benchmarking to see how their outsourcers’ prices compare to the market, but Roehrig says the price benchmarks are limited in their accuracy. And the cards are stacked against customers in the often adversarial benchmarking process. (For more on benchmarking, see “Outsourcers May Try to Prevent Benchmarking.”) “In many cases, the customer gets unhappy and they reach for the benchmarking clause. But the savvy IT service guys know way more than the customer about benchmarking and they contest the benchmark,” says Roehrig. “Everyone ends up really unhappy, except the people who perform the price benchmarking.”

Roehrig advocates a more holistic assessment of outsourcing success or failure. “You need to assess the deal systemically beyond ‘price per widget’ or whether the service provider met the SLAs,” says Roehrig. That would be almost like assessing your physical health by taking your temperature and having you open up and say, “Ah.”

Forrester itself provides such a service it calls a “sourcing deal X-ray,” which includes an onsite assessment of the deal on 150 separate measures. However, says Roehrig, it’s not rocket science. IT leaders can and should try this at home.

The parameters for the outsourcing checkup are based on what Roehrig calls the “key pain points” in outsourcing. According to Forrester, IT services buyers are more likely to be disappointed with the innovation of their provider (42 percent were dissatisfied), their ability to manage change (37 percent), and their business savvy and value (31 percent) than the contractual aspects of the deal (23 percent) or performance against SLAs (15 percent).

Forrester’s Outsourcing X-Ray—Basic Criteria and Weighting

Roehrig contends that outsourcing customers should assess six major areas, asking a variety of questions to score each category, including:

• Business management capability. For example, are there regularly implemented continuous improvement activities related to lower cost of delivery, increased customer satisfaction, productivity improvement and quality improvement?

• Relationship and context. For example, is there a clearly defined overall structure of the management team (provider and customer)? Is there a deal organization chart? Is there alignment between provider and customer? Are account responsibilities clarified? Are there corresponding contacts between the customer and provider organizations?

• Provider market status. For example, what is the provider’s financial strength? How productive is the provider? What does the provider’s portfolio of services look like?

• Expectation matching. For example, to what extent are the provider’s and customer’s expectations of the deal being met for price/cost related to transformation change projects/programs?

• Transformation capability. How well does the provider implement technical change without major disruption and according to plan?

• Delivery management capability. For example, how mature are the provider’s delivery processes? Are they meeting delivery obligations?

At Forrester, each category (and individual criteria listed in each category) is scored on a scale of 1 to 5 and then given an appropriate weight. “It’s soft stuff,” says Roehrig. “But we propose that there are ways to measure things that feel squishy. You may not be able to put an SLA around them. But you can create a meaningful score.”

The process should only take a few days, according to Roehrig, if the right people from the customer and the provider are involved and agree on the methodology and criteria. He admits that can be tricky, particularly if there’s animosity between customer and provider, and notes that, in fact, offshore providers have proven to be more transparent and willing to participate in such deal “X-rays.”

The goal is not just to grade the relationship. “The world doesn’t need more spreadsheets and PowerPoint presentations that don’t get used,” says Roehrig. “Doing an outsourcing deal assessment and not acting on it is a waste of time and money—like going to the doctor, getting his advice and not acting on it. It’s a waste and you feel guilty about it later.” Savvy buyers of IT services use the output to create an improvement program going forward. It’s intended to deal with the most common ailment in the outsourcing world: “We know we’re not satisfied, but we don’t know how to improve it.”

Roehrig says this kind of holistic assessment could have great value even before a deal is signed with an outsourcer. “It would be a really good way to compare bidding providers and get real value from the RFP process,” he says. “The dirty little secret of outsourcing is that most customers get exactly what they ask for.” But he knows it’s more commonly a last resort when the customer or provider becomes truly miserable. A good middle ground would be to perform outsourcing health checkups on a regular basis as part of a continuous improvement plan for outsourcing delivery, deal governance and risk management. “It provides a quantifiable way to document disagreements, but also a way to communicate about the deal,” Roehrig says. “Over time, if done regularly, it can be useful for tracking progress.”