Double Jeopardy

Published: October 15, 1998

Ernie Aschermann is a good software salesman. He knows how to pitch and he knows how to write code—a rare combination of talents in the software industry. On Jan. 16, 1992, he had a rapt audience in the managers of the Oklahoma City Water Utilities Trust, the public corporation that sells water to the citizens and businesses of Oklahoma City.

Desperate to replace its antiquated customer-billing system, the Trust had invited Aschermann and his sales sidekick, Steven “Pax” Darlington, to give a demonstration of a utility-billing program developed by their company, Network Computing Corp. (NCC). In a small conference room at the Trust’s headquarters, Aschermann delivered an impressive presentation. He practically finished the Trust people’s sentences for them, showing off his knowledge of the water utility business and his software know-how.

After building up the Trust’s expectations, Aschermann finally hit a button on his laptop and the screen filled with arcane references to utility-billing functions. “What I want to show you now is Affinity Revision One,” he said. “I’m going to show you this because number one, I don’t want you to go away with the impression that there’s anything that doesn’t exist—any smoke or mirrors.” Then he drove the “trust me” point home with a bit of self-deprecating humor. “What’s that joke about computer salesmen and car salesmen? At least the car salesmen know when they are lying?”

During the disastrous software project that followed, questions arose as to where Aschermann and Darlington fit in that sales spectrum. Allegations of misrepresentation and fraud would land NCC in court a few years later, where a jury decided that the two men might, in fact, have made great car salesmen. Even within the Charlotte, N.C.-based software company, Aschermann’s colleagues had their doubts about his Affinity sales pitch, referring to it from the beginning as “smoke and mirrors.” George Mackie, who was president and CEO of NCC at the time, charged that Aschermann “was stretching the truth to his favor.”

Unstretched, the truth was that Affinity Revision Two (or Affinity 2.1), which the Trust would later buy from NCC for $1.3 million, was far from being fully developed, contrary to the impression left by Aschermann and Darlington that day. The Trust became an unwitting beta site for a desperate, financially troubled software company that needed a big sale to stay afloat.

Ironically, the ensuing catastrophe was not the first of its kind for the Trust. As the NCC salespeople spun their tale in the Trust’s conference room, across town the Trust’s hired-gun lawyers were busily preparing legal documents for the Trust’s case against Peat Marwick Main & Co. (now known as KPMG Peat Marwick LLP), the accounting and consulting firm that had tried and failed to develop a custom-billing program just a few years prior. Like the NCC debacle, that case also involved an unproven software product encased in a sweet, candy-coated sales shell. The complex software overwhelmed Peat Marwick’s well-meaning software developers and the Trust’s expensive new computer system dissolved—like the project—leaving a bitter taste in everyone’s mouth.

That’s why Stacey Davis, utility customer services superintendent for the Trust and the lead business person on the utility-billing system project, decided to videotape (with the vendors’ permission) the sales pitches from NCC and the three other vendors who came to the Trust in early 1992 offering to pick up where Peat Marwick had left off. Davis was prescient, it turned out. A five-minute “highlight tape” from NCC’s presentation became one of the most powerful pieces of evidence during the Trust’s second civil trial against the unfulfilled promises of software vendors and consultants.

The Trust’s 10-year saga of failed software projects and legal wranglings is a cautionary tale for any company contemplating a large, complex software installation using consultants. The story of the Trust shows just how easy it can be for these projects to go horribly wrong and how bittersweet revenge can be when administered through the courts.

Even more disturbing is the fact that Davis and his colleagues at the Trust are not naive. Most had been through software implementations before the big troubles began. In both projects, the Trust took the standard precautions to avoid disaster: clear, detailed requests for proposals (RFPs); exacting contracts that outlined specific deliverables, milestones and dates; full-time project staffers from the business and technology sides of the Trust; and high-level project steering committees that met regularly with the consultants on both projects. Nor were the consultants evil charlatans. Most were dedicated, experienced, talented people. And still both projects ended with unworkable software and angry recriminations on both sides. The Trust’s story is a shocking testament to the ravages of complexity and to the need for businesses to impose radical, almost irrational, levels of control over the management of these projects to avoid major delays, cost overruns and breakdowns.

ACT ONE: A BIG EIGHT BLUNDER

The trust’s painful saga began in 1987, when it resolved to find a replacement for its already obsolete ’70s-era mainframe computer. The Honeywell-Bull system was an expensive technological antique that cost $30,000 or more per month to maintain, according to Davis, and cost Trust staffers precious hours of lost productivity. It was impossible to look up customers by name in the system, for example. Any problems not linked to an account number meant a walk to the research desk, where a massive printout of the Trust’s 170,000 customers sat waiting to waste everyone’s time.

That’s why the Trust decided to bring in Peat Marwick to help figure out what to do with old Bessie. After scanning the software landscape for options, Peat came to the fascinating conclusion that the best software solution was its own. On the surface, Peat’s proposal did indeed look good: a custom-developed software program that would have all the functionality the Trust needed and then some, as well as a fancy new Digital computer, all for $2.6 million. Even better, if Peat could sell the new software to other utilities around the country, the Trust would receive royalties to help defray project costs.

Yet behind the scenes, all was not well with the proposed project, says Ralph Shay, who would become Peat Marwick’s first manager on the project. Shay had heard rumblings within Peat Marwick that the computer Digital had recommended for the project, the VAX 6220, would not be able to handle the large, complex application being proposed to the Trust. The computer choice was critical to the success of the project, because the software did not yet exist and therefore could not be tested to see how much processing power it would consume. But the Trust could not afford to wait before buying a new machine, says Davis, since it did not own a computer powerful enough to handle the software development project, let alone run the finished software. The computer and the software would need to run together seamlessly; the Trust was on a fixed budget and could not afford to swap out the computer for something more powerful (and expensive) once the project got going.

But the Trust had reason to believe that the 6220 was the right machine for the job. Before it became the project consultant, Peat had been hired to coordinate the Trust’s formal search for new hardware and had recommended Digital, which proposed the 6220. Peat Marwick tried to distance itself from that recommendation during contract negotiations with the Trust, but things only got muddier when both sides’ lawyers got involved and inserted a murky accountability clause in the heart of an otherwise clear contract. The Trust bought the 6220 at the same time it signed on with Peat in November 1988. Both parties forged ahead blindly, each assured that the other was legally responsible for any computer problems that arose.

But those who worked on the project for Peat Marwick say the company’s project leadership was more aware of the potential for problems than it let on during contract negotiations. Shay says he made his concerns clear to Larry Lott and Thomas Gorley, his project superiors at Peat Marwick, in a series of meetings in Lott’s Dallas office before the contract with the Trust was signed. “We discussed quite a bit the fact that only one of us on the project team had experience with the combination of hardware and software we were proposing,” he says. “That combination represented an unknown and a risk.”

The new technologies didn’t bother him so much as the computer. “We could learn the new software, but there was a lot of risk in proposing a system that had already been locked into a processor,” he explains. “We felt we could make it work, but there was still a lot of risk.” In sworn depositions taken in 1993, both Lott and Gorley would deny that they had spoken with Shay about the ability—or lack thereof—of the 6220 to run the proposed software prior to signing the contract with the Trust.

Peat was also having problems with its new computer-aided software engineering (CASE) tool, Goldrun, that would generate the computer code for the Trust’s software program. The lone Peat Marwick developer on the Trust project who had experience with Goldrun, Bill Blessing, claims that before the contract was signed, he warned Lott, Gorley and the Trust that Goldrun used bytes like a Sherman tank uses gasoline. During a prior project at an insurance company in Kansas City, Blessing says the application that he helped develop with Goldrun consumed so much processing power that the client had to trade in its Digital 6000 series computer for a 9000 series machine, which was roughly five times as powerful and almost three times as expensive as the 6220. “I knew [the application] would never run on the 6220, and I told many people on both the [Trust] and on the Peat Marwick side my concerns,” says Blessing. Those concerns went unheeded.

To Davis, the hardware dance was irrelevant. He focused on the “acceptance testing” portion of the contract that bound Peat to explicit software The new software posted an absurdly long batch processing time of 91 hours. I told them batch processing would have to be done in Alaska because there they have nighttimes that last six months. -Stacey Davisperformance requirements. Peat was obligated to provide a software program capable of getting information from the 6220 to the terminals of his customer service representatives in two seconds or less, and the system needed to crunch through its load of daily transactions during nighttime batch processing in five hours or less. To Davis, it was Peat Marwick’s responsibility to meet these promises, including buying the Trust a faster computer if that became necessary. Davis says that he confronted Lott and Gorley about Blessing’s warnings but was told not to worry. “They would say the Kansas City system was that system and this is this system and they would reassure us that this was going to succeed,” recalls Davis.

Those assurances became less frequent and less reassuring in April 1989, three months after work on the project began, when Larry Lott sent a letter to the Trust’s MIS director, Jana Bagwell, expressing Peat’s opinion that the 6220 was inadequate for the job. Davis went ballistic. “In steering committee meetings we’d have Gorley saying, ‘That’s a hardware problem, you need to talk to Digital.’ We said, ‘No, it’s your problem. If you want to call Digital in you can, but you are the ringleader here so you are responsible.'”

While the computer issue became public knowledge fairly early in the project, Peat decided not to send a letter to the Trust about the problems with its Goldrun tool, problems highlighted in internal memos written by the project’s technical lead Steward Nazzaro. “The Oklahoma City engagement has a number of requirements which we are finding impossible to meet with the features currently available in Goldrun,” he wrote in February 1989. In December of that year he criticized the “inefficient code” being generated by Goldrun. Although he declined to be interviewed directly for this story, citing a confidentiality agreement signed by the Trust and Peat in their legal settlement, Lott stated in his court deposition that Gorley told him that Nazzaro was “overreacting.”

Nazzaro may have been understating the problem as it turned out. During performance testing in December, the new software posted an absurdly long batch processing time of 91 hours with just one terminal attached to the system (the Trust would need 65 to 80 terminals) and typical information request response times of 11 to 17 seconds. “I told them batch processing would have to be done in Alaska because there they have nighttimes that last six months,” deadpans Davis in his Oklahoma drawl.

Even if Goldrun hadn’t been a bust, there were other signs that the project Peat consultants bickered about the plan for developing and testing the software. But mostly they argued about being put up in a cheesy motel.was in trouble. There was constant turnover among the Peat staff on the project (which Lott characterized in his court deposition as “normal for a project of this size and scope”) and Peat consultants bickered openly with each other in front of Trust staff about the plan for developing the software and the schedule for testing it. But mostly they argued about Peat’s decision to put them up in a cheesy motel. The techies wanted apartments.

Peat Marwick management didn’t do much to quell the project turmoil; of the four project managers assigned to the Trust, three had no prior project management experience.

Things finally came to a head in 1990, when the Trust realized that the project could not be salvaged without going back to the City for more money. Despite well-meaning efforts from all the vendors and the Trust, which agreed to trim its functionality requirements, there was simply no way to bridge the performance gap with the equipment on hand. And no one was willing to chip in the extra $2.5 million for a 9000 Series VAX computer. With all sides clinging to their own versions of project responsibility and accountability, the Trust declared Peat in breach of contract in late 1990 and Peat walked off the project soon thereafter. The Trust would file suit against Peat almost one year later, in June 1991.

The Trust decided to abandon everything that had been developed to that point. “This isn’t like buying a badly built house, where at least you have something to live in despite all the problems,” Davis says. “When Peat Marwick left we had nothing we could use. Nothing.”

Peat would eventually settle with the Trust in August 1993 for $1.8 million, about the cost of Peat’s services. But that did not cover the time spent by the Trust’s staff trying to help develop the new system, nor did it cover the cost of the Digital computer, which was eventually used by another City department. Even with the successful settlement, the Trust still didn’t have the billing system it desperately needed.

ACT TWO: THE VAPORWARE FIASCO

Davis says he was still licking his wounds from the Peat Marwick debacle in late 1991, when he received a phone call from Deniece Chaplin, Digital’s sales representative to the City. “I believe I have found a solution for you,” she said. She told Davis that Digital had just purchased 10 percent of a software company called NCC that offered packaged applications for utilities. This was the first time that Digital had gotten so cozy with a software company and Davis took that as a good omen. He agreed to consider NCC during his renewed efforts to find a utility-billing system.

This time, the Trust decided to write its own RFP, which was lengthy and went into great detail explaining the Trust’s specific functionality and performance requirements. One of the most critical requirements for Davis was that vendors show up with a proven packaged software application—no custom coding or incomplete “vaporware.” NCC was able to present a proven product by affixing the Affinity 1.0 name to a separate program known as Flagship, which was already in use at several U.S. utilities. NCC could then represent Affinity 2.1 as an “incremental upgrade” to Affinity 1.0, rather than an entirely new and separate development effort. By sending the Trust to Flagship reference sites and showing off the Flagship software, NCC avoided the uncomfortable truth about the Affinity product: It was a mere shell of a program that had not been fully tested or installed anywhere in the world.

NCC’s struggle to stay afloat led the company to take desperate measures to win the contract. When Terry McClure, NCC’s project manager, reviewed the Trust’s RFP, he gave NCC management an estimate of 2,800 person hours to customize the base Affinity product (which was still not complete) to meet the Trust’s RFP requirements. He says NCC management then told him and his technical colleagues that “we needed to go back through the [RFP] and not analyze it so closely, I suppose, or words to that effect.” McClure got the message and lowered the estimate by nearly 1,000 hours. He says he believes NCC intentionally “underbid” the project in order to win the contract, something he says is common practice in the cutthroat world of software development.

Davis got his own glimpse of that cutthroat world when a competing bidder sent him a letter warning that no one had Affinity up and running and that one of NCC’s “current” (i.e., Flagship) customers, the public utility of Fayetteville, N.C., kicked NCC out and wrote a new RFP stipulating that “any product proposed must have at least a two-year installation history.”

Davis remembers the “sour grapes letter,” as he calls it. “The software business is a tough business where vendors do a lot of rock throwing,” says Davis. “We called the utilities mentioned and asked what had occurred and we tried to draw a parallel where we could.” He says he couldn’t find many. Besides, he reasoned, the Trust was buying Affinity, not Flagship.

To Aschermann, who still inhabits that cutthroat world as an independent software developer and consultant, pinning the project problems to the Flagship versus Affinity debate is a cop-out. “The basic structure that was started with to develop [Affinity 2.1] was very much the core structure of Flagship,” he says, although NCC developers who worked on the project assert that Affinity was completely new and separate from the Flagship development effort. Aschermann adds that during his videotaped presentation, “I believe it was fairly clearly described that [Affinity 2.1] was not running anywhere although it was being benchmarked at the time.” He is less clear about his videotaped description of Affinity 2.1 code as having been “finished” in November 1991. “I was not in a position to know whether what I was being told by NCC [about the state of the Affinity code] was accurate or not accurate,” Aschermann says. “Was I aware that it was not 100 percent complete? Yes. Was I aware of where it stood exactly? No.”

After signing a $1.3 million contract with NCC in March 1992, it didn’t take long for the Trust to find out exactly where Affinity stood. Nor did it take long to discover that NCC was in such deep financial trouble that it could not afford the computer and people resources necessary to support the development effort for Affinity—which it had agreed to install by April 1993. Soon after the project began, overwhelmed developers at NCC’s Charlotte headquarters began shipping raw, bug-filled Affinity code to Oklahoma City, where the Trust’s programmers took on the burden of testing the rough stuff while McClure and his programmers focused on repairing and installing the code.

Despite the obvious difficulties, McClure remained mum about the true state McClure even denied that the Trust had become a beta site for Affinity. What he didn’t tell the Trust was that in order to meet the standard definition, the Trust would have had to know it was going to be a beta site.of Affinity. In an April 1993 project steering committee meeting, McClure even denied the assertion by City Auditor Brenda Carpenter that the Trust had become a beta site for Affinity. What he knew, but didn’t tell the Trust, was that in order to meet the standard definition, the Trust would have had to know it was going to be a beta site and agree to it before the project started. Asked why he decided not to reveal the truth about Affinity to the Trust, McClure replies, “Just didn’t seem to be in the best interest of the project to indicate that they had bought something that wasn’t there.”

Strange though it may seem now, the company man McClure managed to maintain the respect of the Trust despite all the broken promises from NCC. It had been clear from the beginning that he and his programmers were dedicated to making Affinity work. It’s a good thing, too, because they would need every ounce of sympathy they could muster in September 1993, when NCC’s owner and primary backer, William Devers, decided to pull the financial plug on the troubled Affinity division. “I think it was a Monday,” recalls McClure. “I had flown back from Charlotte to Oklahoma City for another week’s work here on the project. And when I arrived, I couldn’t get into the system. Kerry [Wagnon, the Trust’s new MIS director] informed me that NCC was closing down the Affinity division of the company and the project. So that’s how I officially found out.” McClure had worked for NCC for 17 years. The rest of the Affinity staffers were laid off the same day.

NCC sold the dregs of Affinity to Digital, which promptly shelved it while refusing the Trust’s pleas for help in finishing the project. Left holding the bag once again, Davis knew the Trust could not afford to walk away from yet another pile of dead software. With nowhere else to turn, Davis hired McClure and his former NCC programmers to finish the project and filed suit against NCC and Digital in hopes of recovering some of the costs. McClure and his team would spend another year and a half getting the system to work (at an additional cost to the Trust of $2.6 million) before it finally went live in May 1995.

Davis says he is happy with the system that emerged, except for one major flaw: It lacks a database that would have let Davis run ad hoc queries about customer loyalty, complaints or other vital questions he had about the Trust’s customers. “That was the primary reason we chose NCC over the others,” he laments. What Davis got was a new system that does its job well, but can’t think too much about the people contained in its vast database.

At least it was cheap. In June 1997, an Oklahoma County District Court jury found Digital and NCC both guilty of breach of contract, fraud and misrepresentation, among other charges, and ordered the companies to pay the Trust $4 million and $2 million respectively. Digital paid in full early in 1998, while a battered NCC is making time payments on a reduced settlement of $200,000, according to the Trust’s lawyer, Eric Eissenstat, of the Oklahoma City-based law firm of Fellers, Snider, Blankenship, Bailey & Tippens.

Though the Trust had the last word legally, Davis and the rest of the project team received their fair share of criticism for the two failed software projects. City auditors called for more stringent testing of vendors’ software functionality and computer performance claims in future City projects as a result of the debacles. And outside observers criticize the Trust for trusting too much and not taking a more active role in managing the projects. “You can’t rely on the contract,” says Christopher Hoenig, former director of information management and technology issues at the U.S. General Accounting Office and president of Solutions, a Washington, D.C.-based independent consulting and training firm. “You have to be able to technically assess the project and flag problems early and fix them yourself, if necessary. If you don’t have people internally with the skills to be able to do that, then hire them.”

Davis believes he and his project team did the right thing by focusing onthe agreed upon contractual requirements. “[In both projects] we found that sticking to our requirements was crucial,” he says. “Any time the consultants strayed from meeting their requirements we knew something was wrong and we told them so.” He adds, poignantly: “I also don’t believe it’s good to go around mistrusting everything someone says. That’s just not a healthy way to live.”

Though Davis feels vindicated by the successful legal cases, he looks back on those eight years of lost productivity and broken promises with a kind of pragmatic horror. Through all the pain, he and his staff endured and eventually prevailed. Davis approaches the triple-digit heat of an early June afternoon in downtown Oklahoma City with the same sort of doggedness. “When you live here a while,” he says, “you just learn how to sweat.”

FOLLOW-UP: WHO DO YOU TRUST?

In hindsight, the Oklahoma City Water Utilities Trust should have greeted its consultants’ claims with more skepticism. But the real problem with the handling of the two software projects goes much deeper than that, says Christopher Hoenig, president of Solutions, a Washington, D.C.- based independent consulting and training firm. Though the Trust did most things right during both projects, he says, it erred in one critical area: It ceded project management responsibility to outside consultants and relied on well-written contracts to get out of jams. “This is something that many, many organizations, public and private, fail on,” says Hoenig, who is a former director of information management and technology issues at the U.S. General Accounting Office. “They delegate authority that they themselves should keep, and they allow themselves to be divided and conquered.”

Even though most companies do not have the resources or the expertise in-house to manage complex software projects, they must find a way to retain control over the consultants, he says–even if that means hiring another consultant to help do so.

Public sector organizations are particularly vulnerable in these situations, because they can afford to fail more often and longer than private sector companies. “Public sector organizations don’t go out of business and the customers can’t fire them,” says Hoenig. “You do see these kinds of big failures in the private sector, but they don’t go on for eight years.”

If anyone was a hostage during those eight years, adds Hoenig, it was not the Trust project people, but the Trust’s customers and its customer service staff who spent those years thumbing through paper printouts to look up account numbers. The passing of the years didn’t do much for the Trust’s Honeywell-Bull mainframe, either. One of the system’s more dramatic crashes even made it into the pages of the city newspaper, The Daily Oklahoman, in October 1991, when the Trust could not send out bills for two days. The system limped on while the Trust renewed its search for a replacement.

Stacey Davis, the Trust’s lead business person on the utility-billing system project, is well aware of what the customers and the Trust employees went through, but he is hard pressed to understand how the Trust could have avoided the debacles it faced, particularly with Affinity. “I’ve replayed the whole process in my head time and time again. And if I saw what I saw again, I would probably be more skeptical than I was. But I just don’t know if it would have gotten me to something different,” sighs Davis. “It’s really difficult to deal with a complete untruth.”

Editor’s note: Quotes from sources who were unavailable or declined interviews were excerpted from sworn court depositions obtained by CIO.