25 Questions a Chief Executive Should Ask About Software

Software has been difficult to bring under executive control. In spite of software’s importance to corporate operations, it has remained an intractable technology that causes almost as much trouble as it brings benefits.

Because many CEOs are 40 to 60 years of age, their background and training has included little information about dealing with software. This is also true for many vice presidents of operating units such as manufacturing, sales, marketing and human resources. However, every major corporation has largely automated its financial reporting, billing, marketing and manufacturing. Software is a critical factor for the time-to-market of many new products and may be embedded in the product itself. Ignorance is dangerous.

This article discusses 25 major software topics that CEOs (and other top executives) should understand, to ensure that the software their companies depend upon is an asset and not a liability to the corporations they control. The questions are in five major topic areas:

• Current “hot” topics
• Software usage, user satisfaction and corporate value
• Employee satisfaction and demographics
• The economic impact of software on the corporation
• Competitive analysis: how other companies deal with software

The answers would normally come from the CIO, vice president of information resources, vice president of software engineering or an equivalent position.

The data in this report is drawn from a number of my clients, which include about 150 Fortune 500 companies, several hundred smaller companies, and a number of government agencies and military services. A more complete picture of software issues and best practices can be found in my book, Software Assessments, Benchmarks, and Best Practices (Addison Wesley, 2000).

Current “Hot Topic” Questions

1. How many of our systems are more than 10 years old and becoming obsolete?
2. How useful would agile methods be to us?
3. What is the impact of moving up the SEI capability maturity scale?
4. What is our current level on the SEI CMM maturity scale?
5. What are we doing in regard to ITIL and service-oriented architecture (SOA)?

Most software portfolios are aging, unstable and difficult to modify safely. The average application age is somewhere between 10 and 15 years. One immediate solution is to give the applications “geriatric care” such as renovation and restructuring. Removing error-prone modules is another possibility. Yet another is to outsource maintenance operations, thus freeing up internal resources for new development.

Longer-range solutions involve SOA and the Information Technology Information Library (ITIL). SOA is not yet fully developed and available, though it shows promise. SOA shares some of the issues of enterprise resource planning (ERP), in that the complexity of implementation and deployment is more difficult than anticipated. The ITIL, also a good idea, concentrates on achieving high levels of reliability and availability, plus rapid reaction when issues occur. Within a few years, both SOA and ITIL are likely to have more empirical results than those available in 2007.

For an introduction to these topics, see The ABCs of ITIL and The ABCs of SOA.

Agile methods are based on the premise that small teams of developers in daily contact with users can build working software faster and better than with older methods that start with lengthy and troublesome requirements and design phases. Agile methods have shown impressive results in productivity for smaller applications, and are starting to be used for larger ones as well.

However, the data is still uncertain regarding quality levels of agile applications. Also, agile methods are new enough that little data on long-range maintenance costs is available.

To learn what’s involved in agile methodologies, see The ABCs of Agile Programming.

The next “hot” topic concerns the well-known Software Engineering Institute (SEI), a nonprofit organization that developed an interesting five-point evaluation scale called the “capability maturity model” (CMM), a scoring system that rates companies based on answer patterns to about 150 questions. There is also a newer version, CMMI, or “capability maturity model integration.”

At the most primitive level, CMM Level 1 (accounting for about 75 percent of companies currently), organizations typically have trouble with software cost and schedule overruns, poor quality and cancellations. The top CMM level—achieved by less than 1 percent of companies—ranks the company as “state of the art in all software topics.” Allied with the CMM are the methods of “team software process” and “personal software process” developed by Watts Humphrey.

In general, it requires a year to move from level to level. Ascending to each level may cost as much as $5,000 per capita for the entire software population. For an organization to move from Level 1 to Level 5 may take five years and cost $25,000 per capita for training and retooling costs. These are not trivial amounts.

There is some overlap between the various CMM levels in terms of both quality and productivity. However, as empirical data accumulates, it appears that companies at the higher Levels 2, 3, 4 and 5 often have better quality and sometimes better productivity than Level 1, especially for large applications.

Software Usage, Value and User Satisfaction Questions

1. What are the top complaints people make about our software?
2. Are our customers satisfied or dissatisfied with our service?
3. How many problem reports do we get a year from customers?
4. What are we doing to keep our software useful and valuable?
5. What are our worst current software systems?

Based on surveys by my company, about 70 percent of commercial application users in the United States are fairly well satisfied with both quality and service, as are 65 percent of users of internal MIS applications. For internal corporate applications, slow development schedules leads the list of complaints. Poor quality is the primary complaint for all forms of software. Cost and schedule overruns are common and troublesome.

Today, typical software development teams find and remove only about 85 percent of the defects in applications, which means 15 percent get delivered to users. This is a major industry weakness. However, the best companies are now topping 95 percent in removing defects, and a few projects have even approached 99 percent. For a more complete analysis of U.S. software quality levels, refer to my book, Applied Software Measurement (McGraw Hill, 1996), which summarized data from about 10,000 software projects. (A new edition is in preparation and will be published in 2008, with data current through 2007.)

The topic of less-than-optimal quality leads to the related topic of maintenance for aging software applications. This is because fixing latent defects is the primary maintenance task for several years after an application is deployed.

Starting more than 20 years ago, a new sub-industry appeared of companies and products to provide “geriatric care” for aging software. It is now possible to analyze the structure of existing software and automatically restructure it, using reverse engineering and re-engineering to analyze and convert aging software applications into modern versions. These renovation or “geriatric” tools are proving to be fairly effective.

Companies in many industries have started to reverse the alarming growth in software maintenance and enhancement costs by using these techniques. Some are spending less than 20 percent of their annual software budgets on maintenance; laggards may spend as much as 70 percent. If your company’s maintenance and enhancement costs exceed 50 percent of its total software budget, something may be seriously wrong in your enterprise.

The area of maintenance has been one of the most successful targets of the outsourcing community. Some such vendors are twice as productive as the companies whose software they are maintaining. This is due in part to having trained maintenance personnel, and also because they have developed excellent tool sets. In addition, maintenance outsource groups do not usually split their time between new development projects and legacy application maintenance.

Employee Satisfaction and Demographic Questions

1. What is our current breakdown between software employees and contractors?
2. How many will we need five years in the future?
3. Are there any critical skill shortages that we’re having trouble recruiting to fill?
4. What advantages and disadvantages would we get from outsourcing our software?
5. What are the pros and cons of offshore outsourcing?

The range of outsourcing runs from almost 100 percent to 0 percent as of 2007. However, among most large corporations, outsourcing is in the range of 15 percent to 50 percent of all software workers.

Overall, software professionals in the United States are approximately 2.5 percent of the total workforce. The percentage varies by industry; in banking and insurance, for example, 7 percent to 15 percent of employees have software functions.

More than 75 different occupation groups are associated with software in large corporations and government agencies, including programmers, quality assurance specialists, database administrators, human factors specialists and many more. If your total software staff is larger than 1,000 professionals, and your company cannot identify the occupation groups employed, then something is wrong in your software and human resource organizations.

Large enterprises find that they achieve the best results from judicious usage of software specialists. Maintenance specialists and testing specialists are among the first to be needed, as are database administrators. Software, like medicine and engineering, is too vast a domain for generalists to be expert in every phase.

A number of software specialties have become so popular that shortages exist in the United States. Indeed, for a few specialties it can be necessary to pay “signing bonuses” like professional athletes receive. Among those in short supply today are ERP specialists, Web specialists, and measurement and function point specialists.

One critical topic for which every executive should have current data is the number and cost of canceled software projects. Canceled projects of particular concern are those resulting from excessive cost and schedule overruns, which are severe enough to reduce the value of the software to negative levels.

Of course, there are many business reasons to cancel a software project, such as selling a portion of a business. However, most terminated software projects are canceled because they are so late and so much more expensive than planned that they are no longer valuable. This topic is perhaps CEOs’ chief complaint about software: Large software projects often run late and exceed their budgets.

Function points are becoming the de facto standard for measuring software; the International Function Point Users Group (IFPUG) is the largest measurement association in the world, with 3,000 members from more than 500 corporations and government agencies. A function point is a synthetic metric composed of the weighted totals of the inputs, outputs, inquiries, interfaces and files that users identify as significant in a software application.

Table 1 shows the approximate U.S. distribution of project outcomes, sorted by size into six size plateaus from one to 100,000 function points. Table 1 (taken from Applied Software Measurement) reflects data itself from client surveys and from noting the size of applications that end up in breach of contract litigation.

Table 1: U.S. Software Project Outcome by Size of Project

Because software is important but has been difficult to control, an increasing number of companies are evaluating outsourcing, or are turning over software development and maintenance to companies that specialize in software.

However, not every outsource agreement is satisfactory. Based on my surveys, after 24 months, both parties are generally satisfied in 70 percent of cases, and 15 percent report “some dissatisfaction by client or vendor.” Of the 15 percent remaining, a third are planning litigation or are actively engaged in it.

From process assessments performed within several large outsource companies, and analysis of projects produced by outsource vendors, our data indicates slightly better-than-average productivity and quality control approaches when compared to the companies and industries that engaged the outsource vendors. These results are for development projects. For maintenance projects, the preliminary data indicates that maintenance outsourcing is between 25 percent and 100 percent more productive.

For international outsourcing, the results are more difficult to assemble. However, outsource agreements in India, the Ukraine, the Philippines, Russia and China seem to have more or less the same distribution of results as do the U.S. outsource contracts, although the margin of error is quite high.

Software Economic Impact Questions

1. What is the replacement cost of the software we own?
2. How many of our software projects are canceled and don’t get finished?
3. What are the main reasons for our canceled projects?
4. How much have we spent on canceled projects over the last five years?
5. How many of our applications need immediate replacement?

Corporate portfolio growth rate shows an annual increase of 5 percent to 7 percent in new and changed function points. These are not small numbers. A major multinational manufacturing concern can have upwards of 7 million function points scattered across dozens of software labs and locations. Even a small manufacturing company with fewer than 500 employees can own up to 375,000 function points.

Typically, a manufacturing company builds 40 percent of the function points it uses, leases 30 percent and buys 30 percent. This, too, varies by industry; banks and insurance companies may lease or purchase more than 50 percent of their function points, and build less than 30 percent, because of the ready availability of commercial banking and insurance packages. Smaller companies buy or lease much more software than they build; very small companies may purchase all their software functions.

Also, many companies are migrating toward integrated software solutions such as SAP, Oracle and other ERP software packages; these can replace more than half of in-house applications within many companies. However, ERP installation and tuning is a daunting task with many failures.

Replacement costs vary significantly, but a (U.S.) rule of thumb is that each function point costs $200 to $3,500 to replace; the 2007 average is roughly $1,200. The less expensive replacements are simpler MIS applications.

The life expectancy of software is proportional to its volume. Systems of more than 5,000 function points often run for 10 to 15 years. A rational replacement strategy must be based on both the future business of your company and on your existing portfolio’s size and class variances.

Competitive Analysis Questions

1. Which companies in our industry are best in software?
2. If we spend more on software, what should we spend it on?
3. How does our software productivity compare to our competition?
4. How many of our software projects run late and exceed their budgets?
5. How does our software quality compare to our competition?

Every company should know the best competitors in its own market, ascertained by formal benchmark studies that compare productivity and quality levels within industries.

Historically, software development has been undercapitalized. For software, like all other industries, there is a direct correlation between capital investment and overall productivity. In data collected by the author’s company from Japan, Europe and the United States, it appears that an optimal level of capital investment per software engineer is $5,000 to $10,000 for a powerful workstation, but even more for software tools and support: $25,000 in round numbers.

U.S. productivity for IT software averages eight function points per staff month. Large projects with more than a thousand function points have much lower rates, generally one to five function points per staff month. Small projects, fewer than 100 function points in size, often exceed rates of 20 function points per staff month. Agile projects average about 25 function points per staff month, as do Web applications.

The average number of defects per function point in the United States is four to five, with perhaps 75 percent to 85 percent of those defects found prior to delivery to users. An average of one defect per function point may be present in the software at delivery.

Systems software and military software both have higher defect potentials, but also higher defect removal efficiencies; systems software averages more than 90 percent in defect removal, and military software can average more than 95 percent. To achieve an excellent reputation for high software quality, strive to keep potential defects below 2.5 per function point, and to achieve defect removal efficiencies in excess of 97 percent, for a delivered defect rate of less than 0.075 defects per function point.

For more on software defects, read Quality Doesn’t Just Happen.

There is a strong correlation between high levels of user satisfaction and low levels of delivered defects, which might be expected. However, there is also a strong and surprising correlation between low levels of delivered defects and short development schedules. Since finding and fixing bugs is the most expensive task in software development, leading-edge companies that use sophisticated combinations of defect prevention and defect removal techniques can achieve very short schedules and very high quality levels simultaneously! Short schedules are best achieved by organizations with state-of-the-art quality control methods. Attempts to force short schedules on projects without achieving high quality tends to lead to serious usability problems, litigation and sometimes outright failure.

Just as a medical doctor cannot tell a specific patient about medical problems without a diagnosis, it is not possible to tell a CEO about specific strengths or weaknesses without careful study. However, common software strengths include staff experience levels and capabilities, adequacy of basic tools, high-level language use, and structured methods for development and enhancements. Common software weaknesses include excessive schedule pressure, management skills that rank below technical staff skills, failure to use adequate pretest reviews and inspections, and generally lower levels of capital equipment and support than appears desirable.

Now that the questions and answers have been stated, there is one significant final point: If your company has a large software population, but answers to these questions are difficult or impossible to ascertain, you may need to consider either outsourcing, or a restructuring of your software organization so that topics such as the ones in these questions can be dealt with. In general, leading companies already know the answers to the questions because they have sophisticated measurement programs in place.

If this kind of information is not easily available, you may wish to consider new executive-level positions reporting directly to the CEO and charged with leading software functions into the modern era, such as creation of a vice president of software engineering or a vice president of quality assurance.

Capers Jones is chief scientist emeritus, Software Productivity Research.