25 Questions a Chief Executive Should Ask About Software

This article discusses 25 key questions that CEOs should ask to ensure that the software their companies depend upon is an asset and not a liability to the corporations they control.

Software has been difficult to bring under executive control. In spite of software's importance to corporate operations, it has remained an intractable technology that causes almost as much trouble as it brings benefits.

Because many CEOs are 40 to 60 years of age, their background and training has included little information about dealing with software. This is also true for many vice presidents of operating units such as manufacturing, sales, marketing and human resources. However, every major corporation has largely automated its financial reporting, billing, marketing and manufacturing. Software is a critical factor for the time-to-market of many new products and may be embedded in the product itself. Ignorance is dangerous.

This article discusses 25 major software topics that CEOs (and other top executives) should understand, to ensure that the software their companies depend upon is an asset and not a liability to the corporations they control. The questions are in five major topic areas:

  • Current "hot" topics
  • Software usage, user satisfaction and corporate value
  • Employee satisfaction and demographics
  • The economic impact of software on the corporation
  • Competitive analysis: how other companies deal with software

The answers would normally come from the CIO, vice president of information resources, vice president of software engineering or an equivalent position.

The data in this report is drawn from a number of my clients, which include about 150 Fortune 500 companies, several hundred smaller companies, and a number of government agencies and military services. A more complete picture of software issues and best practices can be found in my book, Software Assessments, Benchmarks, and Best Practices (Addison Wesley, 2000).

Current "Hot Topic" Questions

  1. How many of our systems are more than 10 years old and becoming obsolete?
  2. How useful would agile methods be to us?
  3. What is the impact of moving up the SEI capability maturity scale?
  4. What is our current level on the SEI CMM maturity scale?
  5. What are we doing in regard to ITIL and service-oriented architecture (SOA)?

Most software portfolios are aging, unstable and difficult to modify safely. The average application age is somewhere between 10 and 15 years. One immediate solution is to give the applications "geriatric care" such as renovation and restructuring. Removing error-prone modules is another possibility. Yet another is to outsource maintenance operations, thus freeing up internal resources for new development.

Longer-range solutions involve SOA and the Information Technology Information Library (ITIL). SOA is not yet fully developed and available, though it shows promise. SOA shares some of the issues of enterprise resource planning (ERP), in that the complexity of implementation and deployment is more difficult than anticipated. The ITIL, also a good idea, concentrates on achieving high levels of reliability and availability, plus rapid reaction when issues occur. Within a few years, both SOA and ITIL are likely to have more empirical results than those available in 2007.

For an introduction to these topics, see The ABCs of ITIL and The ABCs of SOA.

Agile methods are based on the premise that small teams of developers in daily contact with users can build working software faster and better than with older methods that start with lengthy and troublesome requirements and design phases. Agile methods have shown impressive results in productivity for smaller applications, and are starting to be used for larger ones as well.

However, the data is still uncertain regarding quality levels of agile applications. Also, agile methods are new enough that little data on long-range maintenance costs is available.

To learn what's involved in agile methodologies, see The ABCs of Agile Programming.

The next "hot" topic concerns the well-known Software Engineering Institute (SEI), a nonprofit organization that developed an interesting five-point evaluation scale called the "capability maturity model" (CMM), a scoring system that rates companies based on answer patterns to about 150 questions. There is also a newer version, CMMI, or "capability maturity model integration."

At the most primitive level, CMM Level 1 (accounting for about 75 percent of companies currently), organizations typically have trouble with software cost and schedule overruns, poor quality and cancellations. The top CMM level—achieved by less than 1 percent of companies—ranks the company as "state of the art in all software topics." Allied with the CMM are the methods of "team software process" and "personal software process" developed by Watts Humphrey.

In general, it requires a year to move from level to level. Ascending to each level may cost as much as $5,000 per capita for the entire software population. For an organization to move from Level 1 to Level 5 may take five years and cost $25,000 per capita for training and retooling costs. These are not trivial amounts.

There is some overlap between the various CMM levels in terms of both quality and productivity. However, as empirical data accumulates, it appears that companies at the higher Levels 2, 3, 4 and 5 often have better quality and sometimes better productivity than Level 1, especially for large applications.

Software Usage, Value and User Satisfaction Questions

  1. What are the top complaints people make about our software?
  2. Are our customers satisfied or dissatisfied with our service?
  3. How many problem reports do we get a year from customers?
  4. What are we doing to keep our software useful and valuable?
  5. What are our worst current software systems?

Based on surveys by my company, about 70 percent of commercial application users in the United States are fairly well satisfied with both quality and service, as are 65 percent of users of internal MIS applications. For internal corporate applications, slow development schedules leads the list of complaints. Poor quality is the primary complaint for all forms of software. Cost and schedule overruns are common and troublesome.

Today, typical software development teams find and remove only about 85 percent of the defects in applications, which means 15 percent get delivered to users. This is a major industry weakness. However, the best companies are now topping 95 percent in removing defects, and a few projects have even approached 99 percent. For a more complete analysis of U.S. software quality levels, refer to my book, Applied Software Measurement (McGraw Hill, 1996), which summarized data from about 10,000 software projects. (A new edition is in preparation and will be published in 2008, with data current through 2007.)

The topic of less-than-optimal quality leads to the related topic of maintenance for aging software applications. This is because fixing latent defects is the primary maintenance task for several years after an application is deployed.

Starting more than 20 years ago, a new sub-industry appeared of companies and products to provide "geriatric care" for aging software. It is now possible to analyze the structure of existing software and automatically restructure it, using reverse engineering and re-engineering to analyze and convert aging software applications into modern versions. These renovation or "geriatric" tools are proving to be fairly effective.

Companies in many industries have started to reverse the alarming growth in software maintenance and enhancement costs by using these techniques. Some are spending less than 20 percent of their annual software budgets on maintenance; laggards may spend as much as 70 percent. If your company's maintenance and enhancement costs exceed 50 percent of its total software budget, something may be seriously wrong in your enterprise.

The area of maintenance has been one of the most successful targets of the outsourcing community. Some such vendors are twice as productive as the companies whose software they are maintaining. This is due in part to having trained maintenance personnel, and also because they have developed excellent tool sets. In addition, maintenance outsource groups do not usually split their time between new development projects and legacy application maintenance.

Employee Satisfaction and Demographic Questions

  1. What is our current breakdown between software employees and contractors?
  2. How many will we need five years in the future?
  3. Are there any critical skill shortages that we're having trouble recruiting to fill?
  4. What advantages and disadvantages would we get from outsourcing our software?
  5. What are the pros and cons of offshore outsourcing?

The range of outsourcing runs from almost 100 percent to 0 percent as of 2007. However, among most large corporations, outsourcing is in the range of 15 percent to 50 percent of all software workers.

Overall, software professionals in the United States are approximately 2.5 percent of the total workforce. The percentage varies by industry; in banking and insurance, for example, 7 percent to 15 percent of employees have software functions.

More than 75 different occupation groups are associated with software in large corporations and government agencies, including programmers, quality assurance specialists, database administrators, human factors specialists and many more. If your total software staff is larger than 1,000 professionals, and your company cannot identify the occupation groups employed, then something is wrong in your software and human resource organizations.

Large enterprises find that they achieve the best results from judicious usage of software specialists. Maintenance specialists and testing specialists are among the first to be needed, as are database administrators. Software, like medicine and engineering, is too vast a domain for generalists to be expert in every phase.

A number of software specialties have become so popular that shortages exist in the United States. Indeed, for a few specialties it can be necessary to pay "signing bonuses" like professional athletes receive. Among those in short supply today are ERP specialists, Web specialists, and measurement and function point specialists.

One critical topic for which every executive should have current data is the number and cost of canceled software projects. Canceled projects of particular concern are those resulting from excessive cost and schedule overruns, which are severe enough to reduce the value of the software to negative levels.

Of course, there are many business reasons to cancel a software project, such as selling a portion of a business. However, most terminated software projects are canceled because they are so late and so much more expensive than planned that they are no longer valuable. This topic is perhaps CEOs' chief complaint about software: Large software projects often run late and exceed their budgets.

Function points are becoming the de facto standard for measuring software; the International Function Point Users Group (IFPUG) is the largest measurement association in the world, with 3,000 members from more than 500 corporations and government agencies. A function point is a synthetic metric composed of the weighted totals of the inputs, outputs, inquiries, interfaces and files that users identify as significant in a software application.

Table 1 shows the approximate U.S. distribution of project outcomes, sorted by size into six size plateaus from one to 100,000 function points. Table 1 (taken from Applied Software Measurement) reflects data itself from client surveys and from noting the size of applications that end up in breach of contract litigation.

Table 1: U.S. Software Project Outcome by Size of Project


1FP 14.68% 83.16% 1.92% 0.25% 100%
10FP 11.08% 81.25% 5.67% 2.00% 100%
100FP 6.06% 74.77% 11.83% 7.33% 100%
1000FP 1.24% 60.76% 17.67% 20.33% 100%
10000FP 0.14% 28.03% 23.83% 48.00% 100%
100000FP 0.00% 13.67% 21.33% 65.00% 100%
Average 5.53% 56.94% 13.71% 23.82% 100%

Because software is important but has been difficult to control, an increasing number of companies are evaluating outsourcing, or are turning over software development and maintenance to companies that specialize in software.

However, not every outsource agreement is satisfactory. Based on my surveys, after 24 months, both parties are generally satisfied in 70 percent of cases, and 15 percent report "some dissatisfaction by client or vendor." Of the 15 percent remaining, a third are planning litigation or are actively engaged in it.

From process assessments performed within several large outsource companies, and analysis of projects produced by outsource vendors, our data indicates slightly better-than-average productivity and quality control approaches when compared to the companies and industries that engaged the outsource vendors. These results are for development projects. For maintenance projects, the preliminary data indicates that maintenance outsourcing is between 25 percent and 100 percent more productive.

For international outsourcing, the results are more difficult to assemble. However, outsource agreements in India, the Ukraine, the Philippines, Russia and China seem to have more or less the same distribution of results as do the U.S. outsource contracts, although the margin of error is quite high.

1 2 Page 1
NEW! Download the State of the CIO 2017 report