A practical hands-on guide for measuring your IT innovation program. Measure performance, competence and strategy to take the black art out of innovation.
By George Chen & Amy Muller
For too long now innovation has been viewed as a black art. Business journals, such as Business Week and Fortune, that
regularly publish their lists of the most innovative companies, select top performers based on surveys of CEO/business executives, not hard data of
performances. Similarly, few managers have the required metrics to make informed decisions about their innovation programs. Therefore, managers of all
types, and IT managers especially (since they are often out of the strategy decision-making loop), have only a vague sense of the innovativeness of their
company and their department; they have little or no means to assess the effectiveness and efficiency of a particular innovation program. Over the last
decade, however, many organizations and their IT departments, have successfully implemented enterprise-wide as well as localized innovation programs.
Along with their successes, and some pioneers’ failures, we now have a more thorough understanding of what and how to measure an organization’s
What to Measure?
Ultimately, innovation is a means to an end — a competence for generating profitable growth opportunities and improving the organization’s
competitiveness. A a holistic measurement system needs to have three perspectives: performance, strength of the competence, and strategic application
of the competence. The performance perspectives report out the “returns” or “results” of an organization’s innovation program(s) while the competence
perspectives report out the ability to envision and implement innovative opportunities. The strategy perspective outlines the criticality and impact of
innovation in the organization’s strategic direction.
To measure the performance, most companies simply track the revenues and profit contributions of their new products and services. Given the
flexibility and capabilities of today’s ERP and/or financial management systems, tracking investment, direct cost, direct revenues, profit margins and other
revenues and profit measures is not highly complicated, as long as the specific products or services can be labeled appropriately. The same is fairly true
for IT organizations. While it’s hard to quantify the total value of the IT department with any precision, it’s substantially more feasible to value the cost and
benefit of specific new programs and initiatives.
Additionally, given the lag time between the envisioning of innovative ideas and commercialization of these ideas, it’s important to measure one’s
innovation pipeline. Specifically, managers should track the value and status of the innovation pipeline.
One way to estimate the value of pipeline is to sum the estimated net present values of all the opportunities in the pipeline, in a similar way as
pharmaceutical companies estimate and track the value of drugs in various stages of development. Specifically, the value of the “innovation pipeline” is the
sum of the values of opportunities, each then discounted by the probability of getting to market and time-value, for example:
VALUE = NPV of Opportunity 1 x Probability of getting to market (based on the stage it’s in)
+ NPV of Opportunity 2 x Probability of getting to market (based on the stage it’s in)
+ NPV of Opportunity 3 x Probability of getting to market (based on the stage it’s in)
+ NPV of Opportunity N x Probability of getting to market (based on the stage it’s in)
A few years ago, a major global commercial bank began an innovation program in its IT organization where employees around the globe share and
nominate improvement opportunities. These opportunities were elaborated and evaluated by the online communities, and the most attractive ones were
then adopted by regional operational units. About a year into the program the bank was able to quantify the value of output by assessing the benefit of
each opportunity implemented. Then, it revised the opportunity submission form to include an estimation of benefits (in magnitude, not a precise number),
and this benefit estimation was further refined as the opportunity went through online elaboration/evaluation and implementation. Now, the bank’s IT
organization reports to its management team the value of its innovation project portfolio and results.
The valuation process of the IT innovation pipeline is similar to the valuation of a new product pipeline, although this bank’s IT innovation pipeline has
fewer stages than the typical new product development pipeline, and the benefit/cost valuation is based on estimation, not a detailed accounting of actual
revenue generated and cost incurred.
Once this calculation is standardized, the change in the total-value of pipeline can be calculated for each time period. Then, the ratio between this
number and the investment in innovation for the same time period is a proxy for return on investment for innovation (innovation ROI). An implication is
that the value of the pipeline and ROI can be increased by adding attractive opportunities in addition to moving opportunities through the pipeline.
Similarly, at any one time, the number of opportunities at each stage of development is a quick overview for the health and growth of the pipeline.
Together, overall revenue/profit contribution from innovation, value of innovation pipeline, status of innovation pipeline and innovation ROI can offer
management a fairly robust view of innovation performance.
Since innovation is a means to an end, savvy CIOs and IT managers go beyond measuring the results of innovation. They also assess the strength or
capacity of this competence. This perspective assesses the extent to which the organization’s skills, processes, culture, and conditions support the
conversion of innovation resources into opportunities for business renewal.
The inputs of this perspective are the preconditions for innovation, i.e. the extent to which an organization’s skills, tools, culture, and values are
adapted to innovation. For example, organizations can assess employees’ personal competence levels, calculate the numbers of employees at various
performance levels, measure employees access to innovation training tools, and benchmark the breadth and quality of training/professional development
The codification of the capability measures the degree to which this capability is embedded and standardized. While the innovation activities will
always rely on the skills and knowledge of the practitioners, many of the methods and techniques can be codified, instead of reinvented at each iteration.
In this area, organizations can measure the extent to which the business and application implementation processes are codified and embedded, and the
level of automation/information system support available.
The outputs of the capability view measure the organization’s success at providing new technology-centric capabilities to the enterprise. For example,
it might measure new competencies (i.e. proficiency with a new technology, adoption of a new application development approach) or newly created
strategic applications (i.e., new pricing application, new supply chain management system). As with the resource view, measurement of both inputs and
outputs is necessary to monitor the extent to which capability view inputs seem to drive capability view outputs.
The input, codification, and output measures should offer management a succinct perspective of an organization’s competence strength in innovation.
This perspective assesses the degree of strategic importance of innovation in the entire organization and IT for the overall organization. It uses
measures such as company’s leadership involvement in innovation activities, the establishment of formal processes to promote innovation, extent of
innovation is embedded into performance metrics, and the criticality of IT in supporting the company’s revenue and profit growth.
Three Common Pitfalls of Innovation Measurements
As more and more organizations embrace innovation as a critical organizational capability, many have also gained operational experiences at
measuring innovation. Amongst these pioneers, they have experienced the following common mistakes:
Setting arbitrary high targets or hurdles for innovation projects. While many organizations have found their innovative products/services
performed better—in terms of profitability or growth—than their traditional products/services, to arbitrarily set higher hurdles or
performance targets could unnecessarily exclude a set of high potential opportunities. For example, one U.S.-based financial services/insurance company
considered using a higher cost of capital for evaluating innovation projects. While it was correct to assume more unknowns or obstacles for innovation
projects, these knowledge gaps didn’t automatically translate into financial risks. Under such a high cost of capital constraint, the company would not have
pursued an innovative opportunity that attracted consumers with small initial deposits but then actively grew these accounts over a long horizon. Given
that this offering had been based on existing products and services, the financial risks were already well analyzed and mitigated so that, subjecting this
opportunity to a higher cost of capital would have caused this company to forgo its now successful new business.
Implication for IT: When implementing a new technology or applying a new application development approach there are many
unknowns. So, the budgeting process should create the appropriate buffer (time, cost) to account for the unknowns. Then, setting the project with higher
hurdle would be “double-counting” the risk.
Placing too much emphasis on any one measure. The three perspectives and a range of measures offer a holistic picture of innovation for
managers, and placing too much emphasis on just one or a few of the measures could lead to unintended consequences. For example, one medical
equipment manufacturer focused on getting as many employees as possible trained on innovation, but the lack of depth by any trainees kept the
organization from fundamentally altering its culture, business processes, incentive system and other enablers of an innovation system. Another organization
measured every employee’s innovation revenue contribution, and this diverted a high amount of resource on debating if commercialized products/services
were innovation. The organization would have been better served if that same resource was applied to improving existing products/services or
envisioning/developing novel products/services.
Implication for IT: Just like enterprises need to apply the entire spectrum of innovation measures in managing its innovation program, IT
managers and executives should utilize the entire spectrum as well.
Separating measures from business decisions. Years after a consumer product manufacturer began embedding innovation across its entire
organization, it successfully trained a critical mass of innovation practitioners and constructed a robust pipeline of innovation opportunities. Along the
way, this company automated tracking of revenues and profits from innovation, and codified a range of business processes. Then, it decided to delegate
the decision of “tagging” innovation to products/services in its ERP to a committee of middle managers. The result was diverting resources from applying
innovation thinking to improve products/services ex ante to debating if specific products/services met the innovation definition ex post of the decision to
pursue these opportunities.
Implication for IT: While estimating benefits and quantifying the value of innovative programs/initiatives help IT and enterprise executive
make informed business decision. But, ultimately it’s a business decision; so, don’t let the desire to archive some performance target supersede business
Smart Metrics for Better Decisions
Innovation is a means to an end. Therefore, organizations should measure across the perspectives of performance, competence and strategy. These
three perspectives help managers understand the results, the capacity for performance, and application of innovation. However, don’t assume a
sophisticated or comprehensive measurement system replaces the need for management deliberation and decision-making. Successful innovation pioneers
develop their innovation measurement system in parallel with management and business decision processes that take advantage of the rich information. In
the end, it’s not just about better knowledge — it’s about better decisions enabled by better knowledge.
George Chen and Amy Muller are Chicago-based Directors of Strategos, a global a strategy and innovation consulting firm. Learn more about
the authors and Strategos here.