Published: October 15, 1998\nErnie Aschermann is a good software salesman. He knows how to pitch and he knows how to write code\u2014a rare combination of \n\ntalents in the software industry. On Jan. 16, 1992, he had a rapt audience in the managers of the Oklahoma City Water \n\nUtilities Trust, the public corporation that sells water to the citizens and businesses of Oklahoma City.\n\nDesperate to replace its antiquated customer-billing system, the Trust had invited Aschermann and his sales sidekick, Steven \n\n"Pax" Darlington, to give a demonstration of a utility-billing program developed by their company, Network Computing Corp. \n\n(NCC). In a small conference room at the Trust's headquarters, Aschermann delivered an impressive presentation. He \n\npractically finished the Trust people's sentences for them, showing off his knowledge of the water utility business and his \n\nsoftware know-how.\n\nAfter building up the Trust's expectations, Aschermann finally hit a button on his laptop and the screen filled with arcane \n\nreferences to utility-billing functions. "What I want to show you now is Affinity Revision One," he said. "I'm going to show \n\nyou this because number one, I don't want you to go away with the impression that there's anything that doesn't \n\nexist\u2014any smoke or mirrors." Then he drove the "trust me" point home with a bit of self-deprecating humor. "What's that \n\njoke about computer salesmen and car salesmen? At least the car salesmen know when they are lying?"\n\nDuring the disastrous software project that followed, questions arose as to where Aschermann and Darlington fit in that sales \n\nspectrum. Allegations of misrepresentation and fraud would land NCC in court a few years later, where a jury decided that the \n\ntwo men might, in fact, have made great car salesmen. Even within the Charlotte, N.C.-based software company, Aschermann's \n\ncolleagues had their doubts about his Affinity sales pitch, referring to it from the beginning as "smoke and mirrors." George \n\nMackie, who was president and CEO of NCC at the time, charged that Aschermann "was stretching the truth to his favor."\n\nUnstretched, the truth was that Affinity Revision Two (or Affinity 2.1), which the Trust would later buy from NCC for $1.3 \n\nmillion, was far from being fully developed, contrary to the impression left by Aschermann and Darlington that day. The Trust \n\nbecame an unwitting beta site for a desperate, financially troubled software company that needed a big sale to stay afloat.\n\nIronically, the ensuing catastrophe was not the first of its kind for the Trust. As the NCC salespeople spun their tale in \n\nthe Trust's conference room, across town the Trust's hired-gun lawyers were busily preparing legal documents for the Trust's \n\ncase against Peat Marwick Main & Co. (now known as KPMG Peat Marwick LLP), the accounting and consulting firm that had tried \n\nand failed to develop a custom-billing program just a few years prior. Like the NCC debacle, that case also involved an \n\nunproven software product encased in a sweet, candy-coated sales shell. The complex software overwhelmed Peat Marwick's \n\nwell-meaning software developers and the Trust's expensive new computer system dissolved\u2014like the project\u2014leaving a \n\nbitter taste in everyone's mouth.\n\nThat's why Stacey Davis, utility customer services superintendent for the Trust and the lead business person on the \n\nutility-billing system project, decided to videotape (with the vendors' permission) the sales pitches from NCC and the three \n\nother vendors who came to the Trust in early 1992 offering to pick up where Peat Marwick had left off. Davis was prescient, \n\nit turned out. A five-minute "highlight tape" from NCC's presentation became one of the most powerful pieces of evidence \n\nduring the Trust's second civil trial against the unfulfilled promises of software vendors and consultants.\n\nThe Trust's 10-year saga of failed software projects and legal wranglings is a cautionary tale for any company contemplating \n\na large, complex software installation using consultants. The story of the Trust shows just how easy it can be for these \n\nprojects to go horribly wrong and how bittersweet revenge can be when administered through the courts.\n\nEven more disturbing is the fact that Davis and his colleagues at the Trust are not naive. Most had been through software \n\nimplementations before the big troubles began. In both projects, the Trust took the standard precautions to avoid disaster: \n\nclear, detailed requests for proposals (RFPs); exacting contracts that outlined specific deliverables, milestones and dates; \n\nfull-time project staffers from the business and technology sides of the Trust; and high-level project steering committees \n\nthat met regularly with the consultants on both projects. Nor were the consultants evil charlatans. Most were dedicated, \n\nexperienced, talented people. And still both projects ended with unworkable software and angry recriminations on both sides. \n\nThe Trust's story is a shocking testament to the ravages of complexity and to the need for businesses to impose radical, \n\nalmost irrational, levels of control over the management of these projects to avoid major delays, cost overruns and \n\nbreakdowns.\n\nACT ONE: A BIG EIGHT BLUNDER\nThe trust's painful saga began in 1987, when it resolved to find a replacement for its already obsolete '70s-era mainframe \n\ncomputer. The Honeywell-Bull system was an expensive technological antique that cost $30,000 or more per month to maintain, \n\naccording to Davis, and cost Trust staffers precious hours of lost productivity. It was impossible to look up customers by \n\nname in the system, for example. Any problems not linked to an account number meant a walk to the research desk, where a \n\nmassive printout of the Trust's 170,000 customers sat waiting to waste everyone's time.\n\n\nThat's why the Trust decided to bring in Peat Marwick to help figure out what to do with old Bessie. After scanning the \n\nsoftware landscape for options, Peat came to the fascinating conclusion that the best software solution was its own. On the \n\nsurface, Peat's proposal did indeed look good: a custom-developed software program that would have all the functionality the \n\nTrust needed and then some, as well as a fancy new Digital computer, all for $2.6 million. Even better, if Peat could sell \n\nthe new software to other utilities around the country, the Trust would receive royalties to help defray project costs.\n\nYet behind the scenes, all was not well with the proposed project, says Ralph Shay, who would become Peat Marwick's first \n\nmanager on the project. Shay had heard rumblings within Peat Marwick that the computer Digital had recommended for the \n\nproject, the VAX 6220, would not be able to handle the large, complex application being proposed to the Trust. The computer \n\nchoice was critical to the success of the project, because the software did not yet exist and therefore could not be tested \n\nto see how much processing power it would consume. But the Trust could not afford to wait before buying a new machine, says \n\nDavis, since it did not own a computer powerful enough to handle the software development project, let alone run the finished \n\nsoftware. The computer and the software would need to run together seamlessly; the Trust was on a fixed budget and could not \n\nafford to swap out the computer for something more powerful (and expensive) once the project got going.\n\nBut the Trust had reason to believe that the 6220 was the right machine for the job. Before it became the project consultant, \n\nPeat had been hired to coordinate the Trust's formal search for new hardware and had recommended Digital, which proposed the \n\n6220. Peat Marwick tried to distance itself from that recommendation during contract negotiations with the Trust, but things \n\nonly got muddier when both sides' lawyers got involved and inserted a murky accountability clause in the heart of an \n\notherwise clear contract. The Trust bought the 6220 at the same time it signed on with Peat in November 1988. Both parties \n\nforged ahead blindly, each assured that the other was legally responsible for any computer problems that arose.\n\nBut those who worked on the project for Peat Marwick say the company's project leadership was more aware of the potential for \n\nproblems than it let on during contract negotiations. Shay says he made his concerns clear to Larry Lott and Thomas Gorley, \n\nhis project superiors at Peat Marwick, in a series of meetings in Lott's Dallas office before the contract with the Trust was \n\nsigned. "We discussed quite a bit the fact that only one of us on the project team had experience with the combination of \n\nhardware and software we were proposing," he says. "That combination represented an unknown and a risk."\n\nThe new technologies didn't bother him so much as the computer. "We could learn the new software, but there was a lot of risk \n\nin proposing a system that had already been locked into a processor," he explains. "We felt we could make it work, but there \n\nwas still a lot of risk." In sworn depositions taken in 1993, both Lott and Gorley would deny that they had spoken with Shay \n\nabout the ability\u2014or lack thereof\u2014of the 6220 to run the proposed software prior to signing the contract with the \n\nTrust.\n\nPeat was also having problems with its new computer-aided software engineering (CASE) tool, Goldrun, that would generate the \n\ncomputer code for the Trust's software program. The lone Peat Marwick developer on the Trust project who had experience with \n\nGoldrun, Bill Blessing, claims that before the contract was signed, he warned Lott, Gorley and the Trust that Goldrun used \n\nbytes like a Sherman tank uses gasoline. During a prior project at an insurance company in Kansas City, Blessing says the \n\napplication that he helped develop with Goldrun consumed so much processing power that the client had to trade in its Digital \n\n6000 series computer for a 9000 series machine, which was roughly five times as powerful and almost three times as expensive \n\nas the 6220. "I knew [the application] would never run on the 6220, and I told many people on both the [Trust] and on the \n\nPeat Marwick side my concerns," says Blessing. Those concerns went unheeded.\n\nTo Davis, the hardware dance was irrelevant. He focused on the "acceptance testing" portion of the contract that bound Peat \n\nto explicit software The new software posted an absurdly long batch processing time of 91 hours. I told them batch processing \n\nwould have to be done in Alaska because there they have nighttimes that last six months. -Stacey Davisperformance \n\nrequirements. Peat was obligated to provide a software program capable of getting information from the 6220 to the terminals \n\nof his customer service representatives in two seconds or less, and the system needed to crunch through its load of daily \n\ntransactions during nighttime batch processing in five hours or less. To Davis, it was Peat Marwick's responsibility to meet \n\nthese promises, including buying the Trust a faster computer if that became necessary. Davis says that he confronted Lott and \n\nGorley about Blessing's warnings but was told not to worry. "They would say the Kansas City system was that system and this \n\nis this system and they would reassure us that this was going to succeed," recalls Davis.\n\nThose assurances became less frequent and less reassuring in April 1989, three months after work on the project began, when \n\nLarry Lott sent a letter to the Trust's MIS director, Jana Bagwell, expressing Peat's opinion that the 6220 was inadequate \n\nfor the job. Davis went ballistic. "In steering committee meetings we'd have Gorley saying, 'That's a hardware problem, you \n\nneed to talk to Digital.' We said, 'No, it's your problem. If you want to call Digital in you can, but you are the ringleader \n\nhere so you are responsible.'"\n\nWhile the computer issue became public knowledge fairly early in the project, Peat decided not to send a letter to the Trust \n\nabout the problems with its Goldrun tool, problems highlighted in internal memos written by the project's technical lead \n\nSteward Nazzaro. "The Oklahoma City engagement has a number of requirements which we are finding impossible to meet with the \n\nfeatures currently available in Goldrun," he wrote in February 1989. In December of that year he criticized the "inefficient \n\ncode" being generated by Goldrun. Although he declined to be interviewed directly for this story, citing a confidentiality \n\nagreement signed by the Trust and Peat in their legal settlement, Lott stated in his court deposition that Gorley told him \n\nthat Nazzaro was "overreacting."\n\nNazzaro may have been understating the problem as it turned out. During performance testing in December, the new software \n\nposted an absurdly long batch processing time of 91 hours with just one terminal attached to the system (the Trust would need \n\n65 to 80 terminals) and typical information request response times of 11 to 17 seconds. "I told them batch processing would \n\nhave to be done in Alaska because there they have nighttimes that last six months," deadpans Davis in his Oklahoma drawl.\n\nEven if Goldrun hadn't been a bust, there were other signs that the project Peat consultants bickered about the plan for \n\ndeveloping and testing the software. But mostly they argued about being put up in a cheesy motel.was in trouble. There was \n\nconstant turnover among the Peat staff on the project (which Lott characterized in his court deposition as "normal for a \n\nproject of this size and scope") and Peat consultants bickered openly with each other in front of Trust staff about the plan \n\nfor developing the software and the schedule for testing it. But mostly they argued about Peat's decision to put them up in a \n\ncheesy motel. The techies wanted apartments.\n\nPeat Marwick management didn't do much to quell the project turmoil; of the four project managers assigned to the Trust, \n\nthree had no prior project management experience.\n\nThings finally came to a head in 1990, when the Trust realized that the project could not be salvaged without going back to \n\nthe City for more money. Despite well-meaning efforts from all the vendors and the Trust, which agreed to trim its \n\nfunctionality requirements, there was simply no way to bridge the performance gap with the equipment on hand. And no one was \n\nwilling to chip in the extra $2.5 million for a 9000 Series VAX computer. With all sides clinging to their own versions of \n\nproject responsibility and accountability, the Trust declared Peat in breach of contract in late 1990 and Peat walked off the \n\nproject soon thereafter. The Trust would file suit against Peat almost one year later, in June 1991.\n\nThe Trust decided to abandon everything that had been developed to that point. "This isn't like buying a badly built house, \n\nwhere at least you have something to live in despite all the problems," Davis says. "When Peat Marwick left we had nothing we \n\ncould use. Nothing."\n\nPeat would eventually settle with the Trust in August 1993 for $1.8 million, about the cost of Peat's services. But that did \n\nnot cover the time spent by the Trust's staff trying to help develop the new system, nor did it cover the cost of the Digital \n\ncomputer, which was eventually used by another City department. Even with the successful settlement, the Trust still didn't \n\nhave the billing system it desperately needed. \n\nACT TWO: THE VAPORWARE FIASCO\nDavis says he was still licking his wounds from the Peat Marwick debacle in late 1991, when he received a phone call from \n\nDeniece Chaplin, Digital's sales representative to the City. "I believe I have found a solution for you," she said. She told \n\nDavis that Digital had just purchased 10 percent of a software company called NCC that offered packaged applications for \n\nutilities. This was the first time that Digital had gotten so cozy with a software company and Davis took that as a good \n\nomen. He agreed to consider NCC during his renewed efforts to find a utility-billing system.\n\n\nThis time, the Trust decided to write its own RFP, which was lengthy and went into great detail explaining the Trust's \n\nspecific functionality and performance requirements. One of the most critical requirements for Davis was that vendors show up \n\nwith a proven packaged software application\u2014no custom coding or incomplete "vaporware." NCC was able to present a proven \n\nproduct by affixing the Affinity 1.0 name to a separate program known as Flagship, which was already in use at several U.S. \n\nutilities. NCC could then represent Affinity 2.1 as an "incremental upgrade" to Affinity 1.0, rather than an entirely new and \n\nseparate development effort. By sending the Trust to Flagship reference sites and showing off the Flagship software, NCC \n\navoided the uncomfortable truth about the Affinity product: It was a mere shell of a program that had not been fully tested \n\nor installed anywhere in the world.\n\nNCC's struggle to stay afloat led the company to take desperate measures to win the contract. When Terry McClure, NCC's \n\nproject manager, reviewed the Trust's RFP, he gave NCC management an estimate of 2,800 person hours to customize the base \n\nAffinity product (which was still not complete) to meet the Trust's RFP requirements. He says NCC management then told him \n\nand his technical colleagues that "we needed to go back through the [RFP] and not analyze it so closely, I suppose, or words \n\nto that effect." McClure got the message and lowered the estimate by nearly 1,000 hours. He says he believes NCC \n\nintentionally "underbid" the project in order to win the contract, something he says is common practice in the cutthroat \n\nworld of software development.\n\nDavis got his own glimpse of that cutthroat world when a competing bidder sent him a letter warning that no one had Affinity \n\nup and running and that one of NCC's "current" (i.e., Flagship) customers, the public utility of Fayetteville, N.C., kicked \n\nNCC out and wrote a new RFP stipulating that "any product proposed must have at least a two-year installation history."\n\nDavis remembers the "sour grapes letter," as he calls it. "The software business is a tough business where vendors do a lot \n\nof rock throwing," says Davis. "We called the utilities mentioned and asked what had occurred and we tried to draw a parallel \n\nwhere we could." He says he couldn't find many. Besides, he reasoned, the Trust was buying Affinity, not Flagship.\n\nTo Aschermann, who still inhabits that cutthroat world as an independent software developer and consultant, pinning the \n\nproject problems to the Flagship versus Affinity debate is a cop-out. "The basic structure that was started with to develop \n\n[Affinity 2.1] was very much the core structure of Flagship," he says, although NCC developers who worked on the project \n\nassert that Affinity was completely new and separate from the Flagship development effort. Aschermann adds that during his \n\nvideotaped presentation, "I believe it was fairly clearly described that [Affinity 2.1] was not running anywhere although it \n\nwas being benchmarked at the time." He is less clear about his videotaped description of Affinity 2.1 code as having been \n\n"finished" in November 1991. "I was not in a position to know whether what I was being told by NCC [about the state of the \n\nAffinity code] was accurate or not accurate," Aschermann says. "Was I aware that it was not 100 percent complete? Yes. Was I \n\naware of where it stood exactly? No."\n\nAfter signing a $1.3 million contract with NCC in March 1992, it didn't take long for the Trust to find out exactly where \n\nAffinity stood. Nor did it take long to discover that NCC was in such deep financial trouble that it could not afford the \n\ncomputer and people resources necessary to support the development effort for Affinity\u2014which it had agreed to install by \n\nApril 1993. Soon after the project began, overwhelmed developers at NCC's Charlotte headquarters began shipping raw, \n\nbug-filled Affinity code to Oklahoma City, where the Trust's programmers took on the burden of testing the rough stuff while \n\nMcClure and his programmers focused on repairing and installing the code.\n\nDespite the obvious difficulties, McClure remained mum about the true state McClure even denied that the Trust had become a \n\nbeta site for Affinity. What he didn't tell the Trust was that in order to meet the standard definition, the Trust would have \n\nhad to know it was going to be a beta site.of Affinity. In an April 1993 project steering committee meeting, McClure even \n\ndenied the assertion by City Auditor Brenda Carpenter that the Trust had become a beta site for Affinity. What he knew, but \n\ndidn't tell the Trust, was that in order to meet the standard definition, the Trust would have had to know it was going to be \n\na beta site and agree to it before the project started. Asked why he decided not to reveal the truth about Affinity to the \n\nTrust, McClure replies, "Just didn't seem to be in the best interest of the project to indicate that they had bought \n\nsomething that wasn't there."\n\nStrange though it may seem now, the company man McClure managed to maintain the respect of the Trust despite all the broken \n\npromises from NCC. It had been clear from the beginning that he and his programmers were dedicated to making Affinity work. \n\nIt's a good thing, too, because they would need every ounce of sympathy they could muster in September 1993, when NCC's owner \n\nand primary backer, William Devers, decided to pull the financial plug on the troubled Affinity division. "I think it was a \n\nMonday," recalls McClure. "I had flown back from Charlotte to Oklahoma City for another week's work here on the project. And \n\nwhen I arrived, I couldn't get into the system. Kerry [Wagnon, the Trust's new MIS director] informed me that NCC was closing \n\ndown the Affinity division of the company and the project. So that's how I officially found out." McClure had worked for NCC \n\nfor 17 years. The rest of the Affinity staffers were laid off the same day.\n\nNCC sold the dregs of Affinity to Digital, which promptly shelved it while refusing the Trust's pleas for help in finishing \n\nthe project. Left holding the bag once again, Davis knew the Trust could not afford to walk away from yet another pile of \n\ndead software. With nowhere else to turn, Davis hired McClure and his former NCC programmers to finish the project and filed \n\nsuit against NCC and Digital in hopes of recovering some of the costs. McClure and his team would spend another year and a \n\nhalf getting the system to work (at an additional cost to the Trust of $2.6 million) before it finally went live in May 1995.\n\nDavis says he is happy with the system that emerged, except for one major flaw: It lacks a database that would have let Davis \n\nrun ad hoc queries about customer loyalty, complaints or other vital questions he had about the Trust's customers. "That was \n\nthe primary reason we chose NCC over the others," he laments. What Davis got was a new system that does its job well, but \n\ncan't think too much about the people contained in its vast database.\n\nAt least it was cheap. In June 1997, an Oklahoma County District Court jury found Digital and NCC both guilty of breach of \n\ncontract, fraud and misrepresentation, among other charges, and ordered the companies to pay the Trust $4 million and $2 \n\nmillion respectively. Digital paid in full early in 1998, while a battered NCC is making time payments on a reduced \n\nsettlement of $200,000, according to the Trust's lawyer, Eric Eissenstat, of the Oklahoma City-based law firm of Fellers, \n\nSnider, Blankenship, Bailey & Tippens.\n\nThough the Trust had the last word legally, Davis and the rest of the project team received their fair share of criticism for \n\nthe two failed software projects. City auditors called for more stringent testing of vendors' software functionality and \n\ncomputer performance claims in future City projects as a result of the debacles. And outside observers criticize the Trust \n\nfor trusting too much and not taking a more active role in managing the projects. "You can't rely on the contract," says \n\nChristopher Hoenig, former director of information management and technology issues at the U.S. General Accounting Office and \n\npresident of Solutions, a Washington, D.C.-based independent consulting and training firm. "You have to be able to \n\ntechnically assess the project and flag problems early and fix them yourself, if necessary. If you don't have people \n\ninternally with the skills to be able to do that, then hire them."\n\nDavis believes he and his project team did the right thing by focusing onthe agreed upon contractual requirements. "[In both \n\nprojects] we found that sticking to our requirements was crucial," he says. "Any time the consultants strayed from meeting \n\ntheir requirements we knew something was wrong and we told them so." He adds, poignantly: "I also don't believe it's good to \n\ngo around mistrusting everything someone says. That's just not a healthy way to live."\n\nThough Davis feels vindicated by the successful legal cases, he looks back on those eight years of lost productivity and \n\nbroken promises with a kind of pragmatic horror. Through all the pain, he and his staff endured and eventually prevailed. \n\nDavis approaches the triple-digit heat of an early June afternoon in downtown Oklahoma City with the same sort of doggedness. \n\n"When you live here a while," he says, "you just learn how to sweat."\n\nFOLLOW-UP: WHO DO YOU TRUST?\nIn hindsight, the Oklahoma City Water Utilities Trust should have greeted its consultants' claims with more skepticism. But \n\nthe real problem with the handling of the two software projects goes much deeper than that, says Christopher Hoenig, \n\npresident of Solutions, a Washington, D.C.- based independent consulting and training firm. Though the Trust did most things \n\nright during both projects, he says, it erred in one critical area: It ceded project management responsibility to outside \n\nconsultants and relied on well-written contracts to get out of jams. "This is something that many, many organizations, public \n\nand private, fail on," says Hoenig, who is a former director of information management and technology issues at the U.S. \n\nGeneral Accounting Office. "They delegate authority that they themselves should keep, and they allow themselves to be divided \n\nand conquered."\n\n\nEven though most companies do not have the resources or the expertise in-house to manage complex software projects, they must \n\nfind a way to retain control over the consultants, he says--even if that means hiring another consultant to help do so.\n\nPublic sector organizations are particularly vulnerable in these situations, because they can afford to fail more often and \n\nlonger than private sector companies. "Public sector organizations don't go out of business and the customers can't fire \n\nthem," says Hoenig. "You do see these kinds of big failures in the private sector, but they don't go on for eight years."\n\nIf anyone was a hostage during those eight years, adds Hoenig, it was not the Trust project people, but the Trust's customers \n\nand its customer service staff who spent those years thumbing through paper printouts to look up account numbers. The passing \n\nof the years didn't do much for the Trust's Honeywell-Bull mainframe, either. One of the system's more dramatic crashes even \n\nmade it into the pages of the city newspaper, The Daily Oklahoman, in October 1991, when the Trust could not send out bills \n\nfor two days. The system limped on while the Trust renewed its search for a replacement.\n\nStacey Davis, the Trust's lead business person on the utility-billing system project, is well aware of what the customers and \n\nthe Trust employees went through, but he is hard pressed to understand how the Trust could have avoided the debacles it \n\nfaced, particularly with Affinity. "I've replayed the whole process in my head time and time again. And if I saw what I saw \n\nagain, I would probably be more skeptical than I was. But I just don't know if it would have gotten me to something \n\ndifferent," sighs Davis. "It's really difficult to deal with a complete untruth."\n\nEditor's note: Quotes from sources who were unavailable or declined interviews were excerpted from sworn \n\ncourt depositions obtained by CIO.