An Audit Tale

Mitre Corp., an independent, not-for-profit company, provides federal agencies with system engineering and information technology expertise. Founded in 1958, the Bedford, Mass.-based company’s more than 4,000 employees support four primary customers: the Department of Defense, the Federal Aviation Administration, the U.S. intelligence community and the Internal Revenue Service.

In June 1994, Mitre began work on a corporate intranet with the goal of transforming the company from a culture that fostered intellectual fiefdoms and internal rivalry to one with an accessible corporate knowledge base and intellectual collaboration. As a provider of intellectual capital to government agencies, company executives felt that collaboration was imperative to Mitre’s long-term success. In May 1995, the corporation debuted its Mitre Information Infrastructure (MII) companywide. Three years later, in an effort to make sure that its goals were being achieved, Mitre began a post-implementation audit of the evolving MII system to capture both its tangible and intangible benefits. (Mitre won a CIO Enterprise Value Award for the system. For more on the MII, see “Common Knowledge,” CIO, Feb. 1, 1999.)

To date, Mitre has invested $7.2 million in the MII, netting an ROI of $62.1 million in reduced operating costs and improved productivity. But financial impact represents only part of the story. According to Al Grasso, vice president and CIO, “Our most important gain can’t be as easily measured-the quality and innovation in our solutions that become realizable when you have all this information at your fingertips.”

In order to gain a complete picture of the system, Mitre conducted an analysis not only of the hard financial benefits but also of the soft benefits the system provided-specifically, how the MII helps employees collaborate more effectively. “To deliver technical excellence, it’s essential for us to share information. It’s how we bring all our resources together to solve problems for sponsors,” explains Mark Maybury, director of artificial intelligence and executive director of the IT division for Mitre. “Our high-level objective with the MII is to make it easier for people to give information to others and to use information from others to solve the next problem that comes along.”

The Hard Benefits

1. Reduced Operating Costs

Government restrictions forbid Mitre from increasing its workload or even making a profit. As a result, Mitre’s perspective on ROI is more qualitative than quantitative. “We have a fixed number of dollars and staff we can deliver because of our unique role as a group of federally funded research centers,” explains Maybury. “Therefore, we need to make people more efficient or save on other indirect costs so that we have more staff to deploy to government projects.”

Assessing reductions in operating costs was fairly straightforward. One of the key measurements Mitre sought to capture was whether the MII enabled the company to apply fewer people more effectively to a task. The Innovation Team, a panel of IT directors responsible for managing Mitre’s IT resources, decided to track efficiencies along operating cost centers. Within each of those cost centers, Mitre examined the impact of moving a number of tasks toward self-service on the MII. For example, with employees able to log on to the MII and update their own human resources records or research routine questions, fewer HR staff were required to support benefits administration. Call logs also indicate that the HR staff now typically handles more sophisticated queries.

In all, the MII has enabled Mitre to save $16.6 million in labor and material costs since 1996. The savings are allocated as follows: human resources and administration ($5.6 million), information systems management ($2.9 million), financial operations ($3.6 million), technical operations ($2 million) and miscellaneous other services ($2.6 million).

2. Improved Staff Productivity

To validate the notion that shifting many mundane activities to the MII would make people more efficient, Mitre focused measurements on three tasks that affect all employees: document management, daily time card submission and purchasing. In the area of document management, the Innovation Team focused primarily on how the MII expedited the dissemination of documents to employees and sponsors. In the past, employees spent tens of minutes a day laboriously converting documents for publication. Today the MII automatically translates PowerPoint presentations, spreadsheets and documents into HTML, and indexes and publishes them so that they’re immediately available companywide. Through surveys and observation, the Innovation Team determined that the technology was saving employees at least five minutes a day that they could devote to other tasks.

By comparing the time spent filling out physical time cards each day with the time it took to submit that data over the MII, the Innovation Team determined that employees saved at least a minute a day with electronic submission. And using a purchasing card online and tracking the status of one’s purchases over the MII, instead of relying on a central purchasing organization, saved the average employee at least two minutes a day. The Innovation Team also factored in improvements in help desk operations and job pricing activity due to the ready availability of key information through the MII. According to time logs, help desk staff saved an average of eight minutes per call; job pricers saved an average of one hour per job. By multiplying these aggregate time savings by the salaries of a conservative three-quarters of the Mitre population ($436,800 per minute), Mitre estimated widespread use of the MII was saving $12.8 million in improved staff productivity.

In the case of the help desk, the Innovation Team also based its estimates on the number of calls handled and the average staff salary-which comes to 56 cents a minute. The value attributed to time saved in performing job pricing was based on 600 jobs priced quarterly and the average salary of staff covering that function ($70,000).

3. Cost Avoidance

Mitre spends tremendous energy and expense complying with federal regulations regarding property management, purchasing and time card submission. In the past, Mitre had had three months’ worth of revenue ($22 million) withheld by sponsoring agencies because of noncompliance. Since it began capturing data on the MII, Mitre has been able to satisfy its contractual record-keeping obligations without any noncompliance penalties.

The Innovation Team also demonstrated that the vast repositories of data housed on the MII would enable the company to realize indirect savings by making it easier and faster to comply with federal regulations. For instance, before the MII, Mitre had an entire department dedicated to tracking computers and peripheral property associated with the dozens of contracts with the Army, Air Force, Navy and other government agencies. The company assigned 600 clerks a year to manually conduct a government-mandated, twice-a-year audit of all the equipment. Now with the MII, every piece of property Mitre purchases is bar-coded and logged into an MII database. Individual employees can log on to the MII and track the status of property they have signed out to them. Government auditors scan the database directly to satisfy their reporting needs. The Innovation Team calculated that using the automating property tracking in lieu of a biannual physical equipment audit saves Mitre $860,000 a year in clerical staff overhead (600 clerks across the company at $9 per hour, wholly dedicated to the auditing process for four weeks a year).

Avoiding costly turnover is another benefit the Innovation Team attributes to the MII. Because the MII facilitates collaboration across geographies, Mitre can leverage the talents of staffers who want to work out of their homes instead of hiring new staff to be onsite. Currently it costs Mitre about $20,000 to process a new employee. The Innovation Team calculated that the MII enabled the company to retain 15 teleworkers, saving Mitre about $300,000 in potential new-hire costs in 1998. According to Maybury, though IT turnover is around 14.5 percent nationwide, Mitre’s turnover rate is significantly less-around 8 percent to 9 percent. He feels part of the credit is due to the MII’s ability to help employees do their job more efficiently. He maintains that the MII lowers job frustration, increases productivity and improves the work environment.

The Soft Benefits

1. Increasing the Knowledge Base

Before the MII, knowledge was a closely guarded commodity within each of Mitre’s three independent business units. Today employees are able to talk knowledgeably about Mitre projects in which they haven’t even participated. To document this shift in culture, the Innovation Team has measured not only the growing volume of contributions to the MII knowledge base but also who has accessed those repositories of information. Every employee is provided with three centrally managed folders: transfer folders for temporary exchange of information, document folders for published knowledge (automatically converted to HTML, moved to a corporate archive and full-text indexed), and homepage and rsum folders for individual profiles. These folders are stored in a distributed file system and serve as the backbone for the corporate knowledge base. By analyzing web logs (time-stamped URLs that identify user navigation patterns and the volume of content accessed on any site) and folder usage, the Innovation Team determined that 35 percent of all Mitre staff publish technical information daily on the MII, and approximately 60 percent retrieve technical information several times a day. Corporation information collections numbered less than 100 in 1995. Today users can access over 300 centrally managed collections.

2. Sharing Expertise

The Innovation Team also uses web logs and transfer folders to measure the growing impact of the MII on collaborative teaming throughout the Mitre organization. The company tracks how frequently employees log on to the online employee directory and individual homepages, as well as how often they access document folders and transfer folders to acquire essential information on customer programs, relevant publications and key contacts.

Web logs and transfer folders indicate that 50 percent of Mitre’s technical project community use the MII as a collaborative development environment several times a day. A survey of all Mitre employees indicates that 91 percent of the respondents felt that the MII had a positive effect on their productivity because it enabled them to find quality information or expertise when needed. Sixty-one percent responded that they could find what they needed faster than ever before. In 1995, web logs showed that the MII averaged 438,000 requests per month. Today the average number of requests per month over the MII exceeds 8 million. According to Grasso, the high traffic reflects an increase in both the depth and breadth of use. He says the user statistics demonstrate that the MII has become an integral part of an employee’s workday, moving beyond routine administrative functions, such as time card submission, to taking advantage of the system’s powerful research tools and information services.

The MII also captures a number of statistics that measure project, department and division collaboration. The financial database allows management to look at what time cards are being charged to which projects. Managers can use that data to generate graphs that show how many employees are working on projects within their organizations, outside their organizations, even outside their operating centers.

Corporatewide, transfer folders are the benchmark for how much collaboration is occurring. The Innovation Team makes inferences by measuring how much information is being put into folders and how much information people are pulling from those folders. Just in the past year, though staffing has remained constant, the number of employees using transfer folders to share project information has grown from 2,000 to 3,000. Additionally, the content of the transfer folders has increased 10 percent every month for the past year. At this point, these transfer folders contain 60,000 documents (over 20GB of information). The Innovation Team also uses the various web logs to determine the average utilization rate of transfer folders throughout the company and to set corporatewide goals. Currently, usage hovers around 30 percent to 40 percent. The goal is to achieve 50 percent or 60 percent across the company. “Not everybody has something that needs to be published all the time,” says Maybury. “But we do want to measure how frequently publishing is occurring. The objective is to get people to share-to give information to others and to use information from others.”

To measure how widespread knowledge sharing is within Mitre, the Innovation Team also tracks who is accessing information to document when individual collections are being retrieved by anyone outside the originator’s department. Executives feel this is an important measurement because it demonstrates the value of cross-project collaboration. A telling example comes from Judith Clapp, assistant to the director of information technologies in the Center for Air Force C2 Systems at Mitre. As the task leader on a U.S. Air Force project, she used the MII to research technology programs germane to her project. She found a relevant briefing published by a staff member in Mitre’s Reston, Va., facility. When contacted, the author contributed additional information and made other contacts that significantly changed the work. “The job was completed in half the time allocated and, more important, it resulted in a better product,” Clapp says.

According to Maybury, the MII’s knowledge repositories are filled with these types of lessons learned. And the benefits continue to mount. Through transfer folder statistics and web log analysis, every month the Innovation Team is able to infer an increasing number of explicit collaborators throughout the Mitre organization.

Conclusion

The value of ready access to intellectual capital was the prime motivator in establishing the MII. With the post-implementation audit, Mitre monitors how well the company is meeting its goals of increasing the knowledge base and fostering cross-organization collaboration. Employee and customer surveys as well as system utilization measurements indicate that the MII is continually expanding Mitre’s ability to exploit its growing corporate knowledge base and provide federal agencies with ever more comprehensive system engineering and information technology expertise.

Have a value methodology you’d like to share and have analyzed? To participate, contact us at casefiles@cio.com. Debby Young is the owner of d’scribe, a freelance writing business based in Framingham, Mass. She can be reached at dscribe@erols.com.

CRITICAL ANALYSIS BY DOUGLAS HUBBARD

Everything Is Measurable

It sounds like MII was a great investment for Mitre. Even the hard benefits alone make MII a clear winner. But this is a commentary on the company’s post-implementation audit, not MII itself, so here are some kudos and some recommendations.

Kudos

Mitre spends the right amount of effort on the audit-more than most would have spent. Companies should endeavor to pay as much attention to measurement as Mitre.

Mitre also makes good use of existing records and random surveys to make measurements. Not all IT departments would have thought of cross-referencing call logs and other records as a measurement method. Some may have labeled “increased query sophistication” as an intangible. Remember, the label of “intangible” always means the labeler just doesn’t know how to measure it.

Recommendations

Mitre could also have avoided considering the “quality and innovation in its solutions” as an intangible. If MII really improves the quality of deliverables, then it should affect customer perceptions and ultimately revenue. Simply ask a random sample of customers to rank the quality of some pre-MII and post-MII deliverables (make sure they don’t know which is which) and if improved quality has recently caused them to purchase more services from Mitre.

Regarding labor savings, many CFOs are skeptical that tiny productivity improvements can add up to big savings, and rightly so. If the time saved simply results in longer lunches, then the value may be nearly zero. We compute a “marginal economic value of labor reduction,” and it generally gives a more realistic value-usually lower-than the traditional “time savings multiplied by salaries” formula.

Finally, I can’t repeat too often the need for businesses to learn how to use probability distributions in IT ROI calculations. Were staff productivity improvements worth exactly $12.8 million? Probably not, but perhaps there is a 90 percent probability that the true value is between $6 million and $18 million. Reporting all values in this way forms the basis of a risk-return analysis and is much more credible. (See “The IT Measurement Inversion,” CIO Enterprise, April 15, 1999, for more information about using distributions.)

The Bottom Line

Regardless of how Mitre could have done the post-audit differently, the really important issue is what Mitre chooses to do with these findings. If the net result of this cost- benefit analysis was only a collective pat on the back, then it was a waste of time. The value of this information comes from how Mitre uses it in future decisions.

Douglas Hubbard is founding partner of Hubbard Ross in Glen Ellyn, Ill. Hubbard Ross uses the scientific and mathematical principles of applied information economics to systematically analyze the economic value of any IT investment. Hubbard is a former IT manager and has more than 10 years’ experience in IT management consulting. He can be reached via e-mail at mailto:dwhubbard@hubbardresearch.com.