Measuring the ROI of Training

Reader ROI

* Learn how an ROI study can help improve your IT training

* Understand the steps involved in an ROI study

* Find out why it is important to measure skills in your organization

NO ONE WOULD OPEN A NEW OFFICE, roll out a new application or even hire a new employee without knowing-not thinking, not guessing, not wishing and hoping, but knowing-they were getting something back. To do otherwise would be bad business. But in the area of IT training, it happens all the time.

The Information Technology Training Association estimates that there are 10 million IT workers in the United States who, according to Stamford, Conn.-based Gartner Group, each cost companies more than $2,000 a year on training. That’s more than a $20 billion dollar market. And in most cases, the justification for spending all that dough is nothing more than the offhand assertion that, yes, employees really like training.

While keeping employees happy in this tight labor market is justification enough for many things, the bigger problem remains: Companies need to be accountable for every dollar they spend. According to Brandon Hall, a Sunnyvale, Calif.-based learning consultant, there is no other workplace issue on which so much money is spent with as little accountability as training. So far, IT training has managed to fly under the radar, but increased shareholder scrutiny combined

Fiscal Responsibility

A 1998 survey by the American Society for Training and Development found that while the importance of IT training has increased in 90 percent of the companies surveyed, only 68 percent predicted their IT training budget would increase as well. Therefore, the only way to meet growing training demands without dramatically increasing your budget is through more effective training. And the most precise way to measure and improve the effectiveness is through a return on investment analysis of training programs.Because there isn’t much incentive to conduct an ROI study if a training program will be approved without one, it is up to the CIO to require employees to justify a training program before he dedicates a slice of the budget to it, Hall says. “If you have poor accountability in your organization, you are wasting money all over the place,” he says. “But you can bring good business metrics to this area, and you can do good studies on the business case for training to make sure your company is getting good ROI on it.”

The Importance of ROI

Allan Wood, CIO of Mellon Financial in Pittsburgh, has sampled this wisdom firsthand. Like at most companies, employee turnover was a large problem for his IT department. “We knew that there was a significant cost in technology staff turnover-both in quantifiable terms and in softer terms,” he says. “It is the cost of recruiting, the cost of bringing the technologist on board, the loss of productivity during the learning stage, but also [costs in] some of the softer [areas] like morale and continuity issues.” Mellon designed a training program aimed specifically at reducing the turnover and, through constant analysis and improvement, dropped its attrition rate from 20 percent in 1997 to 12 percent in 1999 and 10 percent in 2000.

But what Mellon accomplished is not easy, as conducting an ROI of training calls for stringent goal setting and continuous analysis. Mathematically, ROI is a comparison of benefits to cost expressed as a percentage of the original investment. In simpler terms, it is a way of finding if the training met its goals to an extent that justified its cost. According to Greg Wang, director of strategic management for DPT Consulting in Bala Cynwyd, Pa., an ROI study seeks to justify and improve a training program as well as to devise an overall training strategy. There is a misconception that ROI should be left to mathematicians with glasses thick enough to peer through a haze of regression models and partial derivatives. That perception of complexity has deterred many from attempting ROI studies. But, in fact, doing a training ROI is just a fancy way of doing cost-benefit analysis.

And not conducting an ROI analysis of training can cost your company valuable input that’s necessary for improving results. For example, at Lima, Pa.-based Pilot Air Freight, there is currently no mechanism for measuring ROI. Gene Malcolm, Pilot’s senior vice president of administration and information systems, fears that his company’s failure to implement such a measurement has cost the organization the opportunity to hire and train candidates with entry-level skills to replace departed employees, rather than paying contractors as much as $250 an hour to make up the staffing shortage. “If we knew X dollars were being saved by doing this amount of training,” says Malcolm, “we might be more apt to go out and hire additional resources now and get them that training.” Pilot recently added three new IT employees. But maybe, says Malcolm, Pilot would have had a better return by spending its budget on hiring and training six workers with fewer skills and less experience than these highly paid new employees.

Electronic retailer QVC, located in West Chester, Pa., launched such a program last year. The IT department needed to fill eight open positions, and rather than fill them from the outside-its typical practice-it invited employees from across the company to apply for the openings. QVC selected eight employees from the 150 applicants and enrolled them in an extensive eight-week training program covering, among other things, database design, Visual Basic, structured query language and HTML. An ROI study in which the training and productivity costs of the eight employees were weighed against traditional hiring costs found that the cost of training was less than the recruitment costs and the higher salaries outside programmers would have demanded. “We can bring consultants on to write code in a crunch, but we can’t bring in people who know QVC,” says QVC CIO John Link. “We are expecting these people to be high achievers in their career, and we expect them to be very loyal to QVC.” This year the program has grown to 14 employees-well over a third of QVC’s overall IT hires for the year.

Just because a program succeeded at one company, it won’t necessarily work at another; ROI measures too many factors that are unique to the company.

“It’s a pain,” admits Hall. “And that’s why people hardly ever do it.” ROI analysis requires an in-depth understanding of your company’s strengths and weaknesses, strategies, and extensive goal setting. Plus it takes time.With this in mind, Jack Phillips, vice president of Jack Phillips Center for Research (a division of Franklin Covey), based in Birmingham, Ala., suggests that a complete ROI study is appropriate for only about 10 percent to 20 percent of training programs. For instance, if you’re rolling out SAP, it’s obvious that you need to train your IT staff on it. Therefore, doing an ROI process would be pointless. Similarly, analyzing the returns on a presumably low-cost noontime stress reduction seminar probably isn’t worth the effort. On the other hand, a good candidate for an ROI study, says Phillips, is a training program that “probably has a very long life cycle, is very closely tied to operational goals, strategically focused, very expensive and has a high visibility.”

ROI 101

Because ROI requires measurements before, during and after training, finding the ROI of a particular training program is not a discrete event but should be part of an ongoing process. The focus on pretraining measurements also helps align a program with the overall IT strategy. “The first step has got to be to identify my IT needs and goals,” says Martin Bean, president of ProMetric, a Baltimore-based skills assessment organization. “Then what I have to do is compare my existing workforce and the skills that they have got to these goals.” A well-planned training program, he says, should be designed to close the skills gap-and an ROI study is the only way to measure that.

To be certain that the training-and not random, external factors-is responsible for closing the gap, the effects of the training need to be isolated. The easiest way to do this is to measure a behavior in a group of employees before training and again after training. For example, a large financial company that DPT’s Wang worked with had a high turnover rate. A survey of departed employees found that the reason most left was dissatisfaction with the level of training. To address this, the company implemented a skills development program that averaged 80 hours a year per employee. The ROI it was looking for was a decrease in turnover that would represent a savings greater than the cost of the training.

First the human resources department measured the average turnover rate- 23 percent per year-and then calculated the cost of training its 100-person IT department. The average employee wage was $35 per hour, so 80 hours of training for 100 people would cost $280,000. Instructors, classrooms and other costs brought the total to $450,000. The ROI calculation showed that the company reduced turnover by 8 percent, saving $453,000 in related costs for new hires. Ultimately the company used the ROI to reduce training hours because the data showed that employees were actually receiving too much nonproductive, expensive training hours and that less time spent in more focused training actually yielded better results.

While the CIO needs to take the initiative by requiring the ROI study, the task itself can be left to a staff member or, depending on how thoroughly a company wants the study conducted, to an outside consultant. Either way, the CIO will have important data with which to make decisions on current and future training programs.

Here are the five steps the financial company followed to determine the ROI of its training program:

1. Measure the initial reaction to the training. This is the first and most basic level, usually involving a post-training survey assessing the quality of the class. In the case of the financial company, the employees were asked if the material was easy to follow, if the instructor was familiar with the subject matter and if the facility was conducive to learning.

2. Analyze the learning. There are a number of ways to do this, such as testing, simulations and instructor evaluations. What matters most is finding out if the employees actually absorbed the class content. These first two steps are early indicators. Without positive feedback and good marks on the testing, a positive ROI is unlikely.

3. Analyze the skills gained over the long term. The goal of this step is to assess whether on-the-job behavior changed after the training. This data should be collected over a longer period of time. In the example cited by Wang, the year after the training was put in place, turnover dropped 8 percent, giving the company a tangible measure of the training’s effectiveness. During this phase, data is also gathered on how participants have used their new skills.Phillips says this skills analysis can yield valuable data on what worked-and what didn’t-in a training program. “Level three is some of the most valuable data you have,” says Phillips. “If they are not using the skills, we want to find out why. And there we may see disconnects with the environment, lack of support, maybe the wrong skills; it might not be what is needed at this time.”

4. Measure the business impact. This step of the ROI calculation determines the monetary value of the measured change. In the example of the 100-person IT shop cited by Wang, an 8 percent turnover reduction is eight less employees to recruit, train and bring up to speed-a cost of $56,625 per new hire. As a result of the training regimen, the company saved $453,000.

5. Analyze the actual ROI. This phase is really just another way to express the business impact, this time taking into account the program’s cost. In our example, the training cost $450,000 while yielding $453,000 in benefits. To find the ROI, we take the benefits and subtract the total cost, and then divide the result by the total cost. Multiplying this number by 100 yields the ROI percentage. In this case the ROI is .67 percent, or, to put it another way, for every dollar spent the company returned one dollar and two-thirds of a cent.That’s a positive ROI, but hardly the vast savings the company expected. However, just doing the ROI calculation isn’t enough; you have to use the information to make adjustments. In this case, the analysis showed that the lowest turnover rate was among employees who trained 60 to 70 hours annually, and that the turnover rate actually increased for employees with less than 50 or more than 80 hours of training. In other words, not only was it possible to not provide enough training, it was also possible to overwhelm the participants with too much training. Thanks to this data, the CIO decided to reduce training to 64 hours a year, thus cutting costs.

In this case, the CIO used the data to make an informed decision to change the program. And the result was a more efficient and cost-effective training program.

Online Learning

One way many companies have realized significant training cost savings is through online learning. Electronic learning accounts for only a small part of training today, but that is changing. Framingham, Mass.-based IDC (a sister company to CIO’s publisher, CXO Media) sizes the online learning market at $2.2 billion but expects it to grow to a staggering $11.4 billion by 2003.

General Electric is one company going for online training in a big way. In 1998, GE analyzed the efficiency of an orientation course aimed at GE Aircraft Engine customers that required them to travel to GE’s Cincinnati headquarters. The course lasted for three days and cost the company $47 an hour in lost productivity per customer and $1,500 for travel and living expenses for those [customers] flown in. In addition, the training facility needed to be upgraded at a cost of $4.5 million. But rather than do this, GE decided to implement an online version of the training.

An ROI of the online course revealed that employees were able to absorb the same amount of learning as in the onsite training and were able to do this in only three hours versus the three-day course. So the company was able to reduce the amount of time employees spent away from their jobs as well as save money on travel costs.

Russ Mayer, vice president and CIO, e-business and technology solutions, of GE Aircraft Engines, says the online system did lose some effectiveness because of the lack of in-person, face-to-face training but that “the cost savings way make up for it.”

In fact, the savings were so substantial that GE CEO Jack Welch announced last August a corporatewide plan to move 50 percent of all GE’s training online by the end 2001. At GE Aircraft Engines, the IT department has taken on a challenge to reduce its training budget by $100,000 by replacing traditional classroom training with online learning. “Using technology is a key way that we take cost out of the business,” says Mayer. “But it’s not just taking cost out.” Because online learning eliminates days on airplanes and nights in hotels, “we’re looking at this as a key job satisfier so that we don’t have people missing time with their families.”

Another Weapon

While measuring the ROI of training won’t fix all of an organization’s training and budget problems, it is a powerful tool that should be in every CIO’s toolkit. It is an important part of being accountable for your own training budget and for gauging its effectiveness. By finding out how much the IT department is spending on training and then measuring the benefits, both hard and soft, a CIO can not only free up money for other uses, but he can also ensure that the programs are in line with the company’s overall strategy. Given the shortage of IT skills in the marketplace and employees’ demands to keep their skills fresh, failing to measure the effectiveness of training programs is a sure way to cripple an organization. “A CIO has to be very aware of training,” says Mellon’s Wood, “because the people in the organization need to know that it is something that the CIO knows about and cares about.” And doing an ROI of training is one of the best tools a CIO has to ensure the quality of the training being delivered.