Reader ROI\n\nLearn why software is buggier than ever\n\nRead about beta testing best practices\n\nFind out how you can influence the software industrySoftware ain\u2019t what it used to be. According to last year\u2019s President\u2019s Information Technology Advisory Committee report, today\u2019s software often fails "in unpredicted ways." It\u2019s buggy and unreliable, and it just doesn\u2019t do what it\u2019s supposed to.Not that anyone needs to be told that by a federal committee. Everyone already knows about the poor quality of software. It\u2019s even hit the daily newspapers.In April, the Mississippi State Tax Commission filed a lawsuit against Fairfax, Va.-based American Management Systems because the automated tax revenue system that the consultancy built didn\u2019t work without "multiple incidents of both a critical and serious nature," according to the lawsuit. And The Topeka Capital-Journal, a Kansas newspaper, gleefully reported the Mississippi suit by saying that Kansas\u2019s screwed-up tax system relied on the same buggy software.So everybody knows all about this. But CIOs, either because they\u2019re afraid of the vendors they\u2019re dependent on or they fear that any admission of failure will reflect negatively on them, aren\u2019t talking. At least not on the record."I\u2019ll talk about it anonymously, but I don\u2019t want to push the company name out there," says one IT executive who\u2019s frustrated with a vendor and its buggy billing package. "We\u2019ve had some success dealing with them as a beta tester and wouldn\u2019t want to jeopardize that." As a result of this silence, "there\u2019s an amazing lack of detailed examples of serious software failure in the field that can be used to publicize what good and bad [software development] practices are," says Cem Kaner, a Silicon Valley attorney and software quality expert. "CIOs sit on that data." (See "Beta Testing," right, for tips.)While vendors shoulder most of the blame for today\u2019s shoddy apps, some industry watchers argue that a slice of the blame pie belongs to the all-too-quiet CIOs. Scott Gr\u00bfx, technology manager at Windough.com, a Boca Raton, Fla.-based online sweepstakes website, says many of his colleagues "hear the complaints from their users but don\u2019t invest the time to partner with software providers to create a more robust tool. This leaves their end users with information gaps and leaves the software manufacturer with a subpar product." But a handful of CIOs are bucking the trend. They\u2019re making sure they get better software by hopping in on vendors\u2019 test and development cycles and holding out cash until they fix their apps. For these CIOs, upsetting the status quo and sharing their time and know-how with vendors are worth the cost if it means they won\u2019t have to put up with substandard programs, infinite tweaks and endless service calls. It Was the Best of Times..."The software development process used to be a long and complicated thing," recalls Randy Covill, senior analyst of e-commerce strategies and applications at AMR Research, an IT industry analysis company in Boston. As Roger Sherman, former director of testing for Microsoft, explained in his 1995 report "Shipping the Right Products at the Right Time," software quality is defined by three dimensions: reliability, feature set and schedule [the process]. If market conditions change and cause developers to place disproportionate emphasis on any one leg of this stool, the other two can suffer. If being first to market (the opposite of schedule) or feature set takes precedence, warns Sherman, reliability may suffer. For many years, when the market was less volatile, developers produced reliable software because they stuck to safe schedules.According to Covill, the accepted best practices for software development used to call for three to six months of study before a 12-month product cycle even began. After that, developers conducted up to 16 types of tests for functionality, stability, ease of integration and scalability. Then, after the developers deemed the software stable enough to be used in controlled, real-world settings, they shipped it out to beta testers, who reported back to the developers.This beta process benefited both parties. The vendors got a better product because CIOs caught bugs that got past the developers. According to Software Productivity Research, a Burlington, Mass., software consultancy, betas yield some of the highest numbers for fixing errors of omission, commission, ambiguity, speed and capacity in software, compared with 16 other forms of in-house testing. And CIOs benefited by getting the technology first, learning its nuances and gaining leverage with the vendors that allowed the CIOs to ask for and get specific features. Plus, most CIOs received either a discounted or waived application fee for the software....It Was the Worst of TimesBut the old, ideal beta process, says Neil Goldman, director of Internet computing strategies at the Boston technology research organization The Yankee Group, never went according to plan. Some CIOs signed up for betas simply to receive the software in advance for a lower price and never really tested it or informed the vendor about the software\u2019s performance. And many who had every intention of sending feedback became distracted this past decade with priorities like Y2K preparations. Beta-testing responses dropped off so much, says Goldman, that many software shops began holding out on the lower price tag until they received genuine feedback pointing out bugs.Some CIOs simply passed on testing to their subordinates. Ron Furmaniuk, manager of Notes beta administration at Lotus Development Corp., based in Cambridge, Mass., says that of the 4,000-plus testers he manages, most are managers of IS groups, project team leaders, system engineers, analysts and consultants. He cannot find a single CIO or director of IT in his database. And while CIOs became too busy for beta programs, the market changed and developers got busier."A few years back, vendors told customers that the old times of waiting 18 months for a new release were over," recalls Jean-Luc Alarcon, senior vice president of the research reports and interactive\/media resources division of Stamford, Conn.-based consultancy Meta Group. "With the Internet, they said, we\u2019ll deliver you functionality on a regular basis." But, says Alarcon, in their attempts to build apps with more features than their competitors, time to market suffered. So, in their drive to work at Internet speed, the vendors began to cut corners, and 12-month cycles and beta tests went out the window. (Chad Robinson, a senior research analyst at Robert Frances Group, adds that some larger vendors also stopped running beta tests because they didn\u2019t want the details of their releases to leak to their competitors.)As Microsoft alum Sherman predicted, being first to market and feature set took precedence, and reliability took a backseat. A few developers still beta test, but they\u2019re sending out prototypes that they\u2019ve spent only a few months developing and that they know have bugs, says AMR Research\u2019s Covill. If the testers\u2019 feedback is bad, he says, the vendors throw away the prototypes and start over. If the feedback is good (or spare), the app goes to market. "[CIOs] buying new releases are more likely than ever to get versions not far beyond the prototyping stage," says Covill.Hosted software provided by application service providers (ASPs) isn\u2019t much better, according to Goldman. ASP applications, which only live on the ASPs\u2019 servers, can be continually modified, so the ASPs\u2019 don\u2019t have to worry about the robustness of their apps when they release them. There\u2019s never anything to recall. "To get 99.999 percent uptime costs a lot of money," says Goldman, "and they don\u2019t spend a lot of time, money or energy [on testing]. They just want to roll it out." According to Capers Jones, chief scientist at Artemis Management Systems, only 30 percent of U.S. software companies conducted beta tests in 1998, a number that some say has shrunk since then. In the meantime, unpredictable software failures are being caught not by preliminary testers, but by real-time users.Why CIOs Aren\u2019t Beta TestingBrian Bertlin, CIO of Washington Group International, doesn\u2019t beta test. He hasn\u2019t found an app that handles core IT functions so well that it would give his Boise, Idaho, engineering and construction organization a business advantage, and that, he says, would be the only kind of app that would be worth getting in advance and spending extra time testing. "We\u2019re not a dotcom; we don\u2019t need to be on the bleeding edge," he says. First Union\u2019s Jeff Scott says he can\u2019t justify the manpower requirements that testing demands. As the director of advanced technology at the Charlotte, N.C.-based bank, Scott wants to find technology that will improve the bank\u2019s internal communications and external computerized banking offerings. Once upon a time, he organized a group that tested an advance release of Sun Microsystems\u2019 object request broker. And although his team learned the ins and outs of object technology, the bank ultimately decided it wasn\u2019t worth implementing. So Scott\u2019s team had wasted its time. "Our strategy for now," he says, "is to be an early adopter of technology we let others prove."While Bertlin and Scott may be in the majority, a minority of IT executives are fighting the good fight against bad software. CIO talked with three who know that working with and pressuring vendors is the only way to find quality software.Home Depot: La Vida BetaRon Griffin, CIO of Atlanta-based Home Depot, not only reviews applications that he buys before putting them in production, he also often beta tests them. The home goods giant\u2019s IT infrastructure connects more than 700 stores, and Griffin says he\u2019s always looking for new ways to streamline it. Thirty percent of that infrastructure comes from vendors.Griffin believes he is able to use so much foreign, potentially fluky code in his network precisely because he beta tests. He says he always sends new code through his team of testers, but if he does so during beta stages, vendors are more likely to incorporate functionality changes as well as code fixes. "Last year we processed almost a billion customers through our point of sale system," he says. "That probably took 50 billion internal transactions to process. Most applications aren\u2019t designed for the scope and scale of our business. So we\u2019re constantly rearchitecturing and refining." Griffin says he understands why many CIOs use intuition rather than testing to decide if new software will work for their businesses ("Testing is slower," he says), but he thinks that not testing ultimately wastes more time and money. "Organizations that don\u2019t test find they have a harder time getting [the software] into production and their businesses," says Griffin. Once part of the IT staff has tested a vendor\u2019s product, Griffin says they can immediately help train others in the organization. "They get the feeling of ownership through testing," he says. "And it leads to a much higher level of acceptance in the user community."Compaq: On Beyond BetaJoe Batista, the director and chief creatologist of Internet enterprise at Houston-based Compaq Computer, also believes that he avoids bad code by cozying up to vendors. But Batista goes a step further than Griffin and completely immerses himself in a vendor\u2019s production processes. Batista has set up workshops in which he and up to five of Compaq\u2019s engineering, testing, design and business development veterans meet with young software companies\u2019 management teams. They listen to pitches, and they "prod them about their models, fundamental technologies and processes," says Batista. "It\u2019s both collaborative and disruptive."In February 1999, Batista and his crew conducted a session with Conjoin, a Bedford, Mass.-based company that was developing intranet development tools. When Batista heard Conjoin CEO Nick d\u2019Arbeloff tout his company\u2019s Field First intranet publishing tool, Batista realized that it could help him with a project. He had been struggling with an intranet he built for Compaq\u2019s East Coast sales staff. Managing and updating the content on the homepage\u2014timely brochures, spec sheets and customer testimonials were becoming tedious. Field First, which allowed anyone to publish on an intranet with regular word processing applications, might relieve the pressure. But Batista couldn\u2019t just buy Field First because Conjoin hadn\u2019t finished developing it. So Batista offered to help d\u2019Arbeloff finish building the prototype.Batista says this kind of cooperation works because it allows Compaq and its vendors to take chances. "We didn\u2019t have to follow the traditional lines of corporate IT where someone had to be an established vendor with three references and millions of dollars," he says. If the quality of the code or experience of the vendor is questionable, Batista knows he has the leeway to hop in on its development cycle and fill in the gaps.Es\u00bfo: Money TalksSandy Philips, CIO at Es\u00bfo, an e-business consultancy in Berwyn, Pa., doesn\u2019t have the flexibility Griffin and Batista enjoy at Home Depot and Compaq to get involved in vendors\u2019 product cycles (or the clout to convince those vendors to relinquish control of their development efforts). With only four full-time IT workers, he\u2019s too busy keeping his $35 million, 300-plus-person company running smoothly. But he does control a seven-figure budget and the juice that goes with it. Right now, Philips is struggling to integrate into Es\u00bfo\u2019s back-end accounting system a practice management system the company bought last year from a vendor (that he\u2019d prefer not to name). Not only is the product (which Es\u00bfo bought before Philips became CIO) flawed, but the vendor has been slow to fix it."One of the invoices the management system generated was not picking up overtime hours," he says. "We found the bad code and had to massage the data by hand to build invoices. If we hadn\u2019t, it could have cost thousands upon thousands of dollars a month in mis-billing." After discovering the problem, Philips rushed to tell the vendor, but it didn\u2019t respond. So he took the matter into his own hands. "I held all the bills they sent and said, \u2019I won\u2019t pay until they fix the product and I get real performance,\u2019" he says. "Suddenly we started getting a whole lot more voice mails, e-mails and interest." The vendor assigned a full-time account manager to work with Philips.Because of the problems Philips encountered with this vendor, he told another that he wouldn\u2019t pay until he saw performance. "They came to me with set amounts they wanted each month," he says. "I went back to their project plan, pulled out deliverables and said we\u2019ll make milestone payments based on those deliverables, provided they\u2019re good." Now Philips can refuse to pay if this vendor has a code problem that needs to be fixed.Philips has also found vendors to be compliant when he entices them with free advertising at conferences and user group meetings. "I have the power to give them free advertising, whether it\u2019s positive or negative." That, he believes, keeps them on their toes.Power in NumbersPhilips isn\u2019t alone in using user groups as leverage with vendors. Mike Reilly, chief technology officer of J.P. Morgan, a New York City-based financial company, joined a user group to be in close contact with vendors. (See "Clout Helps," Page 112.) Reilly is part of the Open Group, a Reading, England-based user group, one of few not controlled by vendors. It consists of over 200 IT enterprise users and vendors, and encourages IT executives from the likes of Charles Schwab, Visa and the U.S. Department of Defense to vent about software problems. In this manner, Allen Brown, president and CEO of the Open Group, has seen CIOs bring about change. "Often when buyers request solutions, vendors reply that changes are in the next release or that they\u2019re not hearing that from other customers," he says. "But if they hear it right from a group that includes Boeing, J.P. Morgan and Shell, those rebuttals are more difficult to make."Other groups, such as the Research Board, a New York City-based think tank of approximately 100 Fortune 100 CIOs that focuses on business best practices and technology research, exclude vendors. It even excludes CIOs from technology vendors. The Research Board\u2019s researchers visit and interview the vendors, invite major ones to speak and send the vendors the results of CIO surveys. Representatives from standards and certification organizations, such as the Washington, D.C.-based Institute of Electrical and Electronics Engineers Computer Society and The Software Engineering Institute (SEI), a process certification body at Carnegie Mellon University, in Pittsburgh, say they would love for CIOs to join and help shape the software benchmarks they create. "CIOs are some of the most knowledgeable observers, and we\u2019d love to tap their knowledge," says Mike Konrad, a senior member of the technical staff at SEI.Regardless of whether they do so collectively or on their own, David Reid, director of IT at Fazoli\u2019s Management, a Lexington, Ky.-based Italian fast-food company, says it\u2019s imperative that CIOs start talking with their wallets if they don\u2019t have the time or staff to review and test. "We\u2019ve all gone through the cycle of RFP, sales presentations, scripted demos, reference checking and lab testing," he says, "only to discover during implementation significant product shortcomings that were missed in all the previous steps." The answer, he says, is to "be as diligent as you can during the selection process" and remember who pays the bills. "Other than that," he says, "all we can do is wait for evolution and survival of the fittest to move the software industry forward."