Comparing Your IT Shop to Others

CIOs should strive to improve organizational effectiveness and value. One popular way to focus IT performance improvement is to compare an IT organization’s performance to other companies through benchmarking—typically comparing IT costs. But benchmarking on cost is not sufficient and can be dangerous; cost alone does not take into consideration firm-specific factors such as business expectations or context, nor does it give insight into how a firm achieves its numbers; a low-cost organization could simply do “less with less.” To drive performance improvement, CIOs must look at the practices that underlie costs, and they need to ensure they are comparing themselves to firms with similar demands and challenges.

Benchmarking Your IT Shop

• Poorly factor in firm-specific context. Benchmarks can be accurate and precise around commodity technology functions, such as servers per server administrator. But they are more misleading than useful when applied outside these commodity functions. Firm-specific attributes, such as the degree of centralization, or the place of technology in the firm’s operational model, drive how IT operates and thus drive spending. A CIO who cannot explain why his firm has a different cost profile will be on the defensive in all planning and budgeting activities.

• May suggest opportunities, but not explain how to pursue them. Regardless of what benchmark numbers show, you need to know how your management practices compare with known best practices—and practices that are feasible for firms like yours. For example, if your application maintenance spending is higher than the benchmark, do you have strong practices to manage this demand and ensure it reflects business priorities, not just business demand slipping in the back door? If you spend less on architecture activities, does this mean the scope of your architecture program is narrower than best practice firms? Understanding this will tell you whether a “bad” number actually is appropriate, or a “good” number shows missed opportunities.

Process Used for Comparison Determines Usefulness of Results

Firms often use a Balanced Scorecard to manage their performance. The genius behind the Balanced Scorecard is the recognition that managing over time by a single dimension, such as financial metrics, is dangerous. Similarly, comparing your IT organization’s efficiency using a single dimension is misleading, and taking action based on these conclusions is dangerous. An IT organization is a complex system, and so multiple factors and dimensions should be captured and assessed together. For example, the volatility of a market environment, the portion of IT budget applied toward new capabilities, and the practices used to select, resource and manage business initiatives should all be considered as a picture; you could see whether higher spending is due to business demand or to weak project processes—or both. A prudent CIO will take the following approach for comparing an IT organization to others:

First, Find Firms and Organizations With Similar Characteristics

Similar firms are likely to be similar in terms of the expectations they have for and demand they place on IT—and thus IT in these firms provides a useful point of comparison. One way to think about this is as the “business context for IT”—which reflects characteristics that drive demand and expectation. These are not necessarily the ones you might expect and reflect (See Figure 1):

• Cross-industry patterns indicate how IT is utilized. Forrester has found through research and client engagements that factors such as degree of centralization, commonality of business processes, and role of technology in the business model drive patterns of organization, governance and investment. Firms that share these behavioral characteristics are more likely to share similar IT spending and management practices. For example, a firm like Intel might compare its IT with Boeing.

• Industry-specific segmentation is a less useful comparison. Some factors firms gravitate to are less useful for diagnostic benchmarking—with the most common one being industry. Surprisingly, this is only moderately predictive to how IT is utilized—firms may not neatly fit into an industry classification, and may operate very differently from one another—so that a firm like AIG might manage IT very differently from Metlife.

Figure 1: Business—IT factors that influence management practices

Next, Compare Your IT Organization’s Scope and Structure

When you have a basis of comparable enterprises, the next step is to see how your IT organization model compares. Areas to look at include whether there is a designated group for such functions as business relationship management, vendor management or IT marketing/communication, and how centralized or decentralized are functions like architecture, application development or project management. The approach and extent of IT outsourcing is another factor to look at. The reason for this IT organization comparison is to understand:

• Do you have an appropriate IT structure and scope for your firm? Enterprises that tend toward centralized and common processes are best served by IT organizations that centralize functions like planning, resource management and architecture. Firms where technology is core to the firm’s business model are more likely to have business-focused architecture and a stronger client relationship management group. By understanding the interdependencies between firm context and organization, you can tune your organization for improved effectiveness.

• What management practices make sense for your organization model? It is not a given that every IT best practice makes sense for every organization. For example, project portfolio management is a recognized best practice for synchronizing business demand with IT capacity—but should be approached differently if application development is decentralized rather than centralized, or if business relationship managers are senior IT staff.

Then, Compare Budgets Based on Spending Objectives

IT organizations spend their funds on budget categories like staff, software and hardware, and outside services. But comparing what you versus other firms spend on staff without understanding what this staff is being used for provides data that is at best not actionable, and at worst may lead to unjustified actions. Instead, compare your budget allocations based on the purpose for the spending (see Figure 2):

• Separate new investments and planning from ongoing costs. Firms make different decisions around spending on new business initiatives such as new CRM, or to support firm growth, than they do around ongoing operations and support; the former is discretionary, business-driven and very visible, whereas the latter is the result of past investments, existing business expectations and current IT practices. Similarly, spending on planning, enterprise architecture and innovation may reflect a conscious decision to position the firm’s IT for future business needs or cost savings.

• Dig into the context behind the budget numbers. Self-congratulation or blame based on a specific comparison may not be appropriate. For example, if your costs for end-user support are below the benchmark, you can assume that you are running this function fairly lean or your users are not well served. And if your spending for this was above the benchmark, you must evaluate if this is because your users are demanding more support, or if you are not using known practices to reduce this cost.

Figure 2: Budget comparisons tell part of the story

&And Compare Management Practices that Drive IT Value

While the IT budget covers cost, it’s the IT management practices that drive the value your firm gets for its spend. All IT practices at some point should be benchmarked to ensure they are as efficient, effective and of requisite quality. But there is a smaller set of practices that have the greatest impact on the value of the services. Four areas you should examine include (See Figure 3):

• How do you plan and budget? The IT budget for your upcoming year is a reflection of, among other aspects, how you translate business demand into budgets, and how your IT governance structures such as steering committees balance strategic and tactical investments. Firms that wish to drive more IT value find that they first need to address the inputs and processes by which they arrive at their budget by, for example, expanding the charter and membership of an IT executive steering committee.

• How do you allocate budget and resources to projects and services? IT must optimally apply constrained financial and staff resources to business needs, looking at factors ranging from alignment with strategy to project risk and existing demand levels. Firms that gain the most value from IT have strong practices for managing to business cases, scoping and prioritizing initiatives, and tracking these initiatives as a portfolio that reflects these priorities and trade-offs.

• How do you manage projects and service levels? All IT shops should strive to improve quality in project and service delivery. CIOs must investigate project management—such as how you train or source your PM expertise; quality assurance—such as how you use change control; and service management—such as how service-level agreements and operating-level agreements are developed, tracked and communicated.

• How do you ensure business satisfaction and alignment? IT can’t succeed unless it understands and influences business expectations—and when it does, it can ensure satisfaction with IT’s services. Leading firms address this by thinking through their business-IT interaction model and defining roles, such as client relationship managers, to build and maintain the communication channels. And these organizations market IT’s contribution and measure effectiveness and satisfaction over time.

Figure 3: Comparing practices for planning and managing IT

Benchmarking the right way for the right reasons gets the right results

Execs naturally want comparisons, and being open to comparisons gives you a baseline for yourself and an understanding of where you might improve the quality and effectiveness of your organization. Benchmarking of practices is a greater enabler for improvement than benchmarking numbers—but you will likely need to do both. To gain the most useful insights, you should:

• Understand how your context and organization compare to the others. Participate in a multidimensional benchmark, looking at how your enterprise’s demand for, use of and attitude toward IT compared to other firms and organizations. This will help you avoid pursuing strategies that don’t fit your situation.

• Pick the most important areas for improvement. A benchmark will show many gaps with best practices, but you must prioritize what areas you’ll focus on based on size of gap and importance to organizational success. Consider making these changes part of your strategy and operating plans to provide visibility and accountability.

• Use comparison data to improve the perception of IT by your business peers. An appropriately structured benchmark—one that is not focused on cost alone—is a chance to show your peers you are running a well-managed operation. It is also a “teaching moment”—when you can educate them about the factors that drive IT costs and the practices that others use to manage these factors.

• Consider making a lightweight benchmark an annual activity. IT performance improvement must be a continuous focus, not a one-time event. By making a benchmark an annual activity, you instill a management culture based on continuous improvement based on reflection, comparison and adoption of best practices.

Methodology

This report synthesizes the experience of Forrester analysts over hundreds of benchmarking and best-practice assessments. This experience has gone into the methodology behind Forrester’s planning and budgeting benchmark and best-practices service.

By Alex Cullen, with Craig Symons, Bobby Cameron, Paul Warren, Laurie Orlov and Bo Belanger