by Meridith Levinson

SOFTWARE DEVELOPMENT – Let’s Stop Wasting $78 Billion a Year

Oct 15, 200116 mins
Enterprise Applications

Greg Seyk, newly appointed CIO of VisionQuest, had only months to rid his organization of its Y2K bugs. (Y2K, after all, was not a top priority for the Tucson, Ariz.-based national youth services organization, whose mission is to keep kids out of jail, and Seyk had just come on board the previous spring.)

The first thing Seyk did was purchase a Y2K-compliant ERP system?Lawson Insight version 7.1.5?from Lawson Software, a St. Paul, Minn.-based vendor. But just as he and his staff of five got around to testing the software, Lawson released a new version (Lawson Insight version 7.1.6) that included a function for prioritizing bill payments. That function had been promised but never delivered in any of the previous versions, Seyk says.

With just two months to go before the clock struck midnight on Dec. 31, 1999, Seyk didn’t have time to deploy the upgrade, even though the payment prioritization function had been a critical selling point for the 53-year-old CIO. “We had to implement accounts payable, the general ledger, payroll and human resources to make sure they were Y2K compliant. It was no small feat,” says Seyk, who is also a vice president of the private company.

Between November 1999 and July 2001, Lawson released seven new versions of its software to fix bugs or add functionality that had been promised but absent in each previous version. Seyk was outraged. He documented his problems in a series of letters to Lawson executives, met with them on two occasions, and sank a total of $594,974 into software and maintenance to correct the flaws in their products.

And then it dawned on Seyk why the software and support were so bad: That’s the way vendors make money. They push products on the market before they’ve been adequately tested, demand payment up front and then are often not available to deal with the sequelae of poorly performing products. (Lawson officials declined to comment specifically on VisionQuest’s problems with its software. All a Lawson spokesperson would say is that the company is working with VisionQuest in an effort to resolve its concerns. “We are committed to 100 percent customer satisfaction,” says Bev Bergstrom, vice president of communications for Lawson.)

How Bad Software Pays Dividends

CIOs have been complaining about poorly designed and buggy software forever. In a recent survey on, almost half of the 88 IT professionals questioned said they were unsatisfied with both the quality of their business software and the support. (For full survey results, see The problem is a big one: Faulty software costs businesses $78 billion per year, according to Jim Johnson, chairman of The Standish Group, a research company based in West Yarmouth, Mass.

But now many CIOs are beginning to realize that the root of the problem may lie in the economics of the industry. Vendors generate most of their revenues through perpetual licensing agreements, which force CIOs to pay up front for an application. In return, CIOs own the software and the right to use it “in perpetuity.” The problem with this model is that in reality, CIOs are lucky if they can get three years out of a product before vendors release entirely new versions of their software. Vendors further pressure CIOs to buy those new releases by threatening to stop supporting previous releases?a tactic they often take both to cut their tech support costs and to get CIOs to pay again and again for what is essentially the same product.

Another problem with the perpetual model is that CIOs have to fork over an additional 15 percent to 20 percent of what they paid for the software in annual maintenance fees to cover product updates and tech support, according to Chuck Phillips, managing director and software industry analyst at Morgan Stanley Dean Witter. If CIOs want to receive upgrades, patches and access to tech support?as inadequate as it can sometimes be?they have to pay the yearly maintenance fee. Software companies earn a significant amount of cash from these fees. So it’s in the manufacturer’s best interest, at least financially, to make products that need maintenance and that have to be continually improved with successive updates, patches and versions that CIOs pay for up front. In sum, bad software works for the vendors.

There are, of course, other reasons for all the bugs. IT professionals point to a whole litany of causes: bloatware, with all its useless bells and whistles; programmers working in isolation, blissfully ignorant of how people will ultimately be using their software on a daily basis; reusable components that may already contain bugs; an absence of agreed upon professional standards; and developers who take shortcuts to meet deadlines during development.

But a large part of the story may indeed be the way vendors sell software. CIOs are finally waking up to this, and a growing number are demanding that vendors change their business models. A council of IT leaders from a dozen heavy-hitter enterprises convened in August under the auspices of Boston-based analyst company AMR Research, intent on pushing for software industry reform. The group issued a peaceful statement of its desire to “work with” software companies for improvements in quality, delivery reliability and versioning. However, with big names like Becton Dickinson, Boeing, Cabot, General Dynamics and Kraft on the roster, the council has enough weight to change “work with” to “lean on.”

Some other IT users have resorted to more extreme measures?such as withholding payments to put pressure on vendors?but new legislation may soon make it harder for CIOs to employ such brute financial tactics. The uniform computer information transactions act (UCITA) makes it harder for customers to sue vendors and allows vendors to more easily change contract terms. The UCITA has already been passed in Virginia and Maryland and is under consideration in seven other states and the District of Columbia.

Fortunately, there are a host of alternative solutions on the horizon, and a growing number of CIOs are determined to make them a reality. They include renewable licensing agreements, in which CIOs purchase the right to use software for two to three years at about 85 percent of the cost of what they’d pay under a perpetual license. CIOs then have the option to renew the license at the end of the term if they’re happy with the quality of the product and the support. Subscription licensing agreements are similar to renewable licenses, except the term is shorter, lasting about a year, and CIOs rent the software, as opposed to owning it.

Finally, some CIOs are opting to circumnavigate packaged software wherever possible. They’re turning to open-source technologies such as the GNU and Linux operating systems, the Apache Web server and Sendmail e-mail. “People are not involved with [the open-source movement] for profit; they’re involved with it because they want to write good product,” says Bill Lessard, coauthor of NetSlaves: True Tales of Working the Web and a former developer for Prodigy and AOL Time Warner. “If software makers see they are losing money to people going the open-source route, then they will change. Until then, it will be business as usual despite appearances.”

As much as eight years ago, Patricia Wallington, president of CIO Associates and former CIO of Xerox, was envisioning a new method of buying software. “I wanted it to be like a lending library where you could find modules on the Web, buy the ones you were interested in, cobble them together and create your own software,” she says. “We need to rethink the way we deliver software because it is so intransigent.”

Withholding Payment: The Brute Force Option

The economics underlying the software industry?its emphasis on quarterly earnings to impress investors?leads to the pursuit of short-term profits, often at the expense of long-term gains. And this tendency has only been exacerbated by the current market downturn. The revenue of software vendors is predicated on acquiring new customers. That initial sale provides software vendors with their biggest profit. So there is a built-in incentive for vendors to rush a new release of software out the door before it is completely tested and debugged.

Bill Crowell, CIO of Meredith Corp., the $1.1 billion publisher of Better Homes and Gardens, believes it is just this profit motivation that has caused many of the troubles he and other CIOs have had while implementing Oracle 11i. He had purchased Oracle 10.7SC, a client/server-based financial system that handled accounts payable, the general ledger and purchasing functions from the vendor in the spring of 1999, with the assumption that it would be good for at least four years. But in the fall of 2000, Oracle released a new Web-based version, Oracle 11i, and told its customers, including Crowell, that it would be dropping support for all previous releases. Crowell had no choice but to upgrade. (Under pressure from customers, Oracle has repeatedly rolled back the end-support date of these earlier versions. And recently, Oracle officials say they would not immediately be dropping support for older versions.)

Oracle also promised that 11i would include a feature that would automatically enter electronic records of all the purchases Meredith employees had made using their corporate credit cards into the accounts payable or general ledger system either monthly, weekly or daily depending on how the company configured it. Crowell says that when he purchased 11i, the promised functionality was absent. “It wasn’t until about a year later when 11i actually had that capability,” he says. It was an inauspicious beginning.

As soon as he began the upgrade, Crowell found bugs running rampant in the software, like ants scuttling over a piece of fruit. Files were corrupted. Data was lost. Processes didn’t work. Screens froze. “It was just a nightmare,” says Crowell. “We were getting literally dozens of developer patches to this software. Then we were getting patches for patches. The quality was just atrocious.”

One of the biggest bugs bit the interfaces between application components in the financial system. The system didn’t transfer data between accounts payable and the general ledger, between purchasing and accounts payable and between purchasing and fixed assets, Crowell says. The failures were bad enough that had Crowell and his team not been running 11i in a test environment, Meredith would have had to shut down its financial system. It would not have been able to do its accounting or pay its bills until the problems were solved. (Oracle officials declined to comment on either the bugs in 11i or on Meredith’s specific problems with the software.)

“It was clear [Oracle] never tested the interfaces because they flat-out failed the first time. We felt that what was [supposed to be] their general release software was effectively beta,” says Crowell. “There’s no question that they were under pressure from management to be first with a Web-enabled version of their software.”

The 11i implementation was supposed to go live by April 15, but the bugs delayed the implementation by just one month, but only because Crowell’s staff worked 24/7 for four months. The CIO estimates that the bugs cost his company more than $100,000; he had to pay for contractors to help with the nine-month implementation, and he wasn’t able to put staff on other pressing projects.

So how did he muddle through this debacle with Oracle? “We didn’t pay them, for one. We owed maintenance of $300,000 to $400,000, and we just didn’t pay it. We said, ’We’re holding on to it until you get this thing up and running,’” Crowell says.

But he thinks it wasn’t so much the money that got Oracle to fix the bugs in 11i as it was the brute force he and his project leaders applied in dealing with the vendor. They called Oracle daily to see if the company was making headway resolving their problems. They also forced Oracle to give them contacts in the development group so that they could ask developers directly for help rather than going through the support team.

Crowell blames economic forces for the problems with Oracle 11i. “They’re trying to move so fast to get the product into the marketplace that they’re not adequately testing and debugging their software,” he says. If Oracle had waited six months before releasing 11i and taken that time to test the application, he says, the upgrade would have gone off without a hitch. “Overall we’re very pleased with the new application, but if Oracle thinks they’re the Lexus of the software industry, [after] what they’ve done to their customers, you feel you’ve bought a Dodge De Soto,” Crowell concludes.

Renewable Subscriptions: Use Now, Pay Later

Crowell believes that the new renewable and subscription arrangements that are becoming more prevalent in the software world would have ameliorated the problems he ran into while deploying 11i. Under a subscription model, in which he would have paid less up front, Crowell would have had more leverage. Also, it would have given Oracle a greater financial incentive to please Crowell. In fact, Crowell plans to start buying software from Microsoft on a subscription basis in two years, once he finishes receiving all the upgrades he paid for two years ago. To him, the renewable model makes sense. It’s the way his own publishing industry works. “It’s a subscription. We know when the revenue is coming in. We can plan our business around it,” he says. “And we deliver a quality product every month. The [vendors] need to think about delivering quality every month and a business model that allows them to do that.”

Microsoft, in fact, announced that it would begin offering a brand-new subscription license this month for its operating systems and software, including Microsoft Windows Professional and Microsoft Office Professional. The Enterprise Agreement Subscription, as the new license is called, is a major departure from the perpetual model. CIOs will now lease the software under subscription licenses. While CIOs see the potential benefits of the subscription model, many are uncomfortable with the specific terms Microsoft is offering. For instance, Microsoft is requiring that customers pay a hefty annual fee even before new upgrades are released.

Microsoft has also introduced a new and more controversial twist to its perpetual Open and Select licensing agreements. These programs pressure CIOs to upgrade to new versions such as Windows XP by Feb. 1, 2002, at a discounted rate. If they don’t, they will have to pay twice as much to upgrade after that date. (For more information on this controversy, go to “Looks Can Be Deceiving” on

Mark Grove, CIO of AmericasDoctor, a pharmaceutical services company based in Chicago, says software vendors that require perpetual licenses with constant upgrades are not serving their customers. “Any vendor who’s doing that is trying to force the customer to follow the vendor’s business model,” Grove says. “When they tell a customer that he has to upgrade at a certain time, they’re forgetting that their customers have their own business cycles and busy seasons that they have to work within.”

After his experience with Lawson, VisionQuest’s Seyk is also considering buying software from vendors on a renewable or subscription basis when it is offered. He likes the idea of not paying for the software entirely up front. “Once you give [vendors] the cash and the software doesn’t perform, you have no leverage,” he says. “If [CIOs] could say, Sure, you’ll get 10 percent now and 10 percent after each quarter, and a year from now you’ll get it all if [the software] works,” that would be a way to hold the vendor accountable. “It gives [the vendor] a financial incentive to make sure the product works,” he adds.

Open Source, Open Sesame

Some CIOs believe open source could liberate businesses from their dependence on the dysfunctional software-making ma-chine. Linux now boasts important implementations at companies including Shell Oil, hotel franchiser Cendant, networking giant Cisco and retailer Burlington Coat Factory, while the Apache program is by many accounts the most widely used Web server software today. The prevalence of these open-source packages will only continue to grow, with big IT vendors such as IBM providing hardware with Linux and Apache bundled in.

Raymond Dury, co-CIO of Ameritrade Holding, says open-source software, because of its community-style development method, has flexibility that commercial software lacks. The packaged software market doesn’t offer the solutions he needs to support his company’s thousands of daily online transactions. To enable customers to make trades faster, hold more trades in their portfolio and get real-time quotes, for example, Ameritrade’s website needs enormously robust software code?code that can support tens of thousands of simultaneous connections with sub-second responses.

Instead of buying software and additional hardware to extend the speed and computing power of his website, Dury is looking to configure his own software using open-source code to support all of those transactions.

He also wants to replace three separate software components that communicate back and forth when a customer wants to execute a trade with one piece of proprietary software built using open-source code to consolidate these functions. “We have separate pieces of software communicating with each other, trying to do one process,” Dury says. “Instead we could use a piece of open source to consolidate these separate functions under one piece of software.”

Dury has long been a fan of open source. He used it to develop a secure e-mail application while serving as the vice president of operations at Netdox, a Deloitte and Touche venture in 1997. “Those folks [involved in the open-source movement] are very knowledgeable, very good at what they do, and they’re producing really great code,” says Dury.

Open source would give CIOs the flexibility to build exactly what they need, he says. However, many are skeptical about using it because the technology is immature, and it’s hard to find programmers who know how to write the code and maintain it. When you buy software from a vendor, you can always turn to its help desk, however incompetent. With open source, you’re on your own.

Dury acknowledges that open source requires seasoned in-house IT staffs?who know how to build and integrate their own systems?and that not all companies can afford to hire this kind of talent. But he likes open source for the same reason that Crowell and Seyk like the renewable options to perpetual licensing agreements. “If you increase competition [in the market] because there’s quality in open-source software and because it’s low cost, some folks will move toward it,” Dury says. “As a result, vendors are probably going to have to increase their quality and reduce their costs.”