CIOs reveal how they measure vendor success

The global IT market is projected to reach $3.7 trillion in 2018, a year-over-year increase of 4.3 percent, according to Gartner.

Underpinning this massive figure are countless everyday interactions between CIOs and the vendors who serve them — interactions that lead to collaboration or friction, understanding or confusion, success or failure.

Strategic IT leaders leave nothing to chance. They rely on specific tools and methods to measure vendor effectiveness and to determine continued investment.

In this edition of CIO Quick Takes, global IT leaders share how they assess whether or not an existing vendor relationship is worthwhile.

Tom Anfuso, Senior Vice President and CIO, National Life Group

Management guru Peter Drucker famously said, “culture eats strategy for breakfast,” and there are few places this is as visible as in the tech sector. In my 25 years in the IT world, I have watched and sometimes participated in the rise and decline of technology titans, often on the back of corporate culture change. At its simplest, corporate culture is how employees treat people — both colleagues and the clients of their firms. As Herb Kelleher of Southwest Airlines observed, “Culture is what people do when no one is looking.”

Everyone in the tech industry has a great PowerPoint deck (Keynote if they are from Silicon Valley). They often have technology and talent to back up the slick visuals. But what about their corporate culture? Firms routinely pitch their solutions to me and my team, during which they often say little or nothing about the culture of their own company. That’s a big omission because, as our CEO, Mehran Assadi, often says, “Without culture, an organization has no soul.” You should expect no better experiences or values from a firm than its employees experience working there.

What does the supplier’s culture value and reward? Internal innovation? Transparency? Service-orientation? Or do they really just care about margin contribution and EBITDA? How does their culture show up in the way that they work, in their incentives, and to influence their most important decisions? Does the account rep or sales engineer even know what the mission and values of their own company are? If a strong culture isn’t at the front and center of their thinking, ask yourself why, and to what effect for customers like you?

I am fortunate to work at a company that takes culture very seriously. Our company’s mission is literally keeping our promises. That’s more than a tagline; it’s a strategic commitment and ethical Due North. Our values are even simpler — Do good. Be good. Make good. Consequently, dishonesty, double-dealing and anything less than complete respectfulness and service orientation by an employee is a ticket out the door. We certainly don’t just throw this culture out the window when it comes to our suppliers. We provide and expect a level of transparency, ethics, responsiveness and straight-talk that is probably unusual for our tech suppliers. We accept that we might be an outlier. But in every tech supplier relationship, you are going to have ups and downs. It is usually a healthy, client-centric culture on the part of the supplier, not burn down charts or patents, which will get you through those rough patches.

So, when the culture of a supplier or prospective supplier isn’t aligned with ours in a meaningful way, we part ways with them. We have ended relationships with major tech providers largely, and sometimes entirely, for this reason. In many cases, they have solid or even leading technology. But if a supplier routinely demonstrates little or no cultural alignment with our organization, we will cut the cord as quickly as our contracts allow. On the flip side, when a supplier shows great cultural fit with our firm, we both recognize and reward that good behavior, including formal awards. 

Our suppliers are at least collaborators, if not genuine partners. Their culture alignment with ours unleashes superior people performance for competitive advantage. We will not compromise that form of competitive advantage for any vendor — even a technologically superior one.

Culture matters on your team, your division, your company, and the suppliers on whom you rely. Make a point of incorporating cultural dimensions into your vetting, selection and ongoing review of your suppliers. Know when there is a bad fit, and accept that culture almost never changes quickly. When your suppliers are routinely missing the mark on ethics, service or values, don’t expect miracles just because a senior supplier exec parachutes in and promises better outcomes. That supplier’s culture will take years to change, if it does at all. In the end, culture fit will trump Magic Quadrants, roadmaps, market share and demos. I encourage CIOs and their teams to give supplier culture the focus it deserves.

Mike Skinner, Executive Vice President and CIO, EURPAC Service, Inc.

We use two methods to evaluate vendor effectiveness: One for vendors of commercial off-the-shelf (COTS) products and services that are readily available from any number of vendors, and another for vendors of the unique products and services we acquire as we build and maintain distinct value propositions in our marketplace.

We expect COTS vendors to operate in a fashion that is not only effective (they get the job done), but they must operate in a fashion that is highly efficient (competitive pricing and ease of doing business). Most of our vendors in this category are large publicly traded organizations, which provides us the visibility we require as to their fiscal health and strategic direction.

Unique products and service vendors are a different story. Some of these vendors are large, but several are smaller, privately held organizations. In this case, we evaluate the vendor’s effectiveness through a combination of objective and subjective measures.  The objective measures help us evaluate the corporate efficacy of the vendor (their business health), as well as their investments in their tools and talent. We will align only with vendors that demonstrate fiscal health and sound organizational planning.

By design, three subjective measures are less “black and white.” Although cost of product and service remains a factor, all three of these are subjective measures of the vendor’s commitment to our organization. First, is the vendor willing to take the time to learn about our business and our value proposition? Second, will the vendor support us on good days and bad? Will they be there when we need them most? And third, are our strategies aligned? Are we likely to grow in directions that create synergy or separation?

Evaluating current and prospective vendors is a key to success of all organizations, and every CIO. Take the time to know your vendors — you are counting on them!

Joel Jacobs, VP, CIO & CSO, MITRE Corp.

Setting clear expectations ahead of time makes for a more practical and expected vendor evaluation process. At MITRE, there are several factors that influence how we evaluate our vendors’ effectiveness.

A good contract

It starts with a good contract that includes performance-based incentives constructed on well-defined metrics and SLAs. These vary depending on the service being provided, but they’re important to have, since they provide a basis for incentives and penalties. The best contracts are written with the flexibility to allow for service enhancements mid-contract and, if necessary, streamlined extensions.

Partnership with vendors

Unsurprisingly, our most successful vendors are often our longest tenured. Because the vendors know our business and environment, a level of trust has been established. These trust (but verify) partnerships enable several benefits, including:

• Agility to adapt to changing conditions 
• Willingness to quickly jump in during a crisis
• Open and direct dialog on performance and their product/service plans
• Participation and influence in customer advisory councils
• Stability of account representatives

Partnership with internal customers

When appropriate, we encourage key internal customers to be part of the vendor engagement team. Additionally, we often survey the customers to ensure they are realizing the value intended in the contract. The results are typically part of the vendor performance metrics and can play a significant part in measuring effectiveness.

Measurement at regular intervals

It’s important to meet at regular intervals (typically monthly) to review the performance metrics, discuss awards or penalties, and identify trends or any required changes to the service. 

Escalation processes

Having an escalation path with the vendor can help if the account representative isn’t working out or if they need help within their organization to prioritize our issues.

These processes and engagements help to keep the service levels and customer expectations on track, as well as strengthen our partnership with the vendor. Setting parameters at the outset and building in flexibility and contingencies helps the vendor deliver what we want, while taking the ambiguity out of measuring vendor effectiveness.

Ken Piddington, CIO & Executive Advisor, MRE Consulting, Ltd.

In today’s fast-paced world of technology change and disruption it is critical for an organization’s success that they have an effective vendor management program.  Vendor management programs are not one size fits all and must be right-sized for the organization’s needs and culture. 

For vendors to deliver the real value required by companies, they must become true partners. The vendor management program should focus on creating real partnerships to drive value for the company. A key component of a program is measuring the effectiveness of your partners. This is critical for understanding where value is being delivered and where value isn’t being delivered from your partners. The program I’ve been successful with has two primary components to help monitor partner effectiveness: Partner Categorization and Partner Assessment.

For Partner Categorization, all partners are put into categories based on their products/services value to our organization. This is done because the company we buy laptops from is not as critical to our business success as the provider of a mission-critical application, and needs to be managed differently. The categories that have worked for me have been called Strategic, Emerging, Tactical, and Legacy. 

The Partner Assessment is a scorecard that breaks down a partner’s performance into 11 sections. The sections we use are Strategic & Corporate Fit, Business Relationship Maturity, Project Management Capabilities, Quality & Timeliness of Deliverables, Cost Effectiveness, Quality of Support, Quality of Staff, Operational Capabilities, Information Security, Contractual Compliance, and Customer Satisfaction.

Within each section are specific items to be measured, and the sections are weighted based on the partner’s category and type of product/service provided. The scoring is done by the members of my team that own the relationship with partner. For Customer Satisfaction, the various users throughout the company of the partner’s products/services also provide input for the scoring.

Matt Lasmanis, Vice President of U.S. Technology, GSK

At GSK, our tech team covers a wide spectrum of activities, from leading-edge digital work to back-office financial systems. And almost all of our efforts involve vendors or third parties to some degree.

Across our traditional projects and services, the classic foundational measurements of time and cost are important for ensuring operational fidelity and evaluating execution. 

Quality is prime, of course. We have found that standardizing our software quality metrics, such as how we define and measure defect rates, is useful when comparing performance across projects and development partners.  

In our more experimental work, where we are trying to test a hypothesis, such as a new business concept, we focus initially on whether we are getting the results we anticipated. This requires an up-front definition of what success looks like. For example, we may try a software product in a very limited setting to understand if it will deliver the necessary business outcomes and performance on a small scale. This approach helps us assess whether the product and partner are going to be able to scale the solution enterprisewide. 

Regardless of the type of project, one tool we have found extremely useful is an after-action review. These reviews help us extract key learnings and ensure we apply them to future engagements. In one of our recent after-action reviews on a relatively small project, we had an insight into how we can significantly improve how we assess third-party resources for software development projects. We are now embedding this assessment as a standard for all our projects.