EVERYONE KNOWS the rules for managing software projects. You\u2019re supposed to test the code. Frequently. You\u2019re supposed to take ownership of the process, have a business sponsor and keep an eye on developments. You\u2019re supposed to be alert to problems and potholes, and you\u2019re supposed to be ready to step in to fix the glitches as they arise. Or kill the project if it looks hopeless.The only problem is the rules don\u2019t help.In our Jan. 15 "Bugs!" story about companies struggling with poorly written software, we suggested that CIOs frequently test and take ownership of the process. The year before, CIO ran "Another Trip to Hell" (Feb. 15, 2000), and about a year before that we published "To Hell and Back" (Dec. 1, 1998)?both stories about software project failures and the lessons they teach: "Test, test and test again," "Ownership is essential" and "Don\u2019t let a doomed project run on." And lest you think this is a relatively recent development, you should know that in 1967, Ken Kolence, cofounder of Boole & Babbage, a pioneering software testing company, published a paper for the first NATO Software Engineering Conference outlining some best practices for the new field of software engineering. It featured instructions on how to test code, assign a manager to own a project and kill projects that were going nowhere. That\u2019s right. The accepted wisdom for managing software development hasn\u2019t changed in almost 35 years. And what has the accepted wisdom achieved?Not much. We Begin with a Litany of FailuresA landmark 1994 white paper, "The Chaos Study," published by The Standish Group, a West Yarmouth, Mass.-based consultancy, reported that just 16 percent of software projects succeed. The rest either failed (31 percent) or were challenged (53 percent)?a term encompassing cost and time overruns and missing features. Standish turned "Chaos" into a longitudinal study, collecting case studies (30,000 and counting) and each year publishing success, failure and challenge rates. Its 2000 report, "Chaos in the New Millennium," is about as encouraging as its cover art, which includes the grim reaper rising through clouds, brandishing his scythe.Outright failures, Standish reported, have declined from 40 percent to 23 percent during the past five years, but challenged projects swelled from 33 percent to 49 percent in the same period. That\u2019s bad because challenged projects often are more painful than projects that simply fail, like peeling off a bandage slowly instead of quickly. And they\u2019re often just failures-in-waiting, dallying dismally until the patience (or the money) for getting them right runs out.Put another way, almost three-quarters of all software development in the Internet era suffered from one or more of the following: total failure, cost overruns, time overruns, or a rollout with fewer features or functions than promised. CIO could run "Still Yet Another Trip to Hell" next month. The article could include Nike\u2019s glitch with i2\u2019s inventory software, which prompted Nike CEO Phil Knight to wail, "This is what we get for our $400 million?" It could mention Sobey\u2019s, a Canadian grocery chain that in February suddenly and publicly canned an SAP project. (You could practically hear the sonic whoosh of $50 million following the press release into the trash bin.) Or it could tell the tale of a 100-year-old management consultancy that, according to an inside source, spent tens of millions on a failed ERP deployment, then tried and failed again after sinking another $8 million into the black hole.But what would be the point? A litany of failures and maddeningly familiar advice?Nike needed to test more; Sobey\u2019s needed a project sponsor; the consultancy should have killed the project?simply restates what everyone already knows. The problem isn\u2019t with the advice; the advice is sound. The problem is bigger. The problem is the development methodology itself. The problem, says Martin Fowler, programmer and cofounder of The Agile Alliance, is that companies construct software much the way people build bridges. (And Fowler should know. His wife, Cindy Chabot, is a structural engineer. Bridges are what she builds.)Why Building Software Is Not Like Building a BridgeA bridge begins with a blueprint based on mathematical certainties and predicted levels of tolerance that never change. The blueprint is followed precisely. Changes are costly and therefore anathema. The tools used to build the bridge are standard and don\u2019t change during construction. The materials are familiar and behave predictably.Because of all that, we\u2019re pretty good at building bridges. They rarely fall down. But bridges are built to do only one thing: connect two pieces of land so that people can cross between them without falling. Software tries to do many things, and many of them have never been tried. And the tools used to build the software change continuously.Therefore, as long as software engineers act like bridge builders, they are doomed to fail. And the cost of that failure is beginning to rise."If this were the pharmaceutical indus-try, we\u2019d be killing people," says Joshua Greenbaum, analyst at Enterprise Applications Consulting in Daly City, Calif. Greenbaum tracks ERP, the clay pigeon of software failure stories. "There\u2019s no reason to tolerate this level of failure. It tars the software industry. And frankly, it\u2019s bad for the CIO."Check that. It used to be bad for the CIO. Now, in today\u2019s economic climate, it\u2019s disastrous. Nike issues an earnings warning, and Knight shuffles his executive team. Sobey\u2019s ditches a grocery application and its CIO, Bradley Jardine. The consultancy ousts most of its C-level crew.Desperate to avoid the scapegoat\u2019s horns, some technology executives are finally beginning to take up arms against this sea of failure, redefining how software is built. They call it Agile Development, a disciplined, minimalist approach that\u2019s both elegant and arduous, and maybe IT\u2019s best hope to avoid "Yet Another Trip to Hell."We, the Programmers...The nicest thing Fowler says about traditional software development is "It\u2019s great except that it doesn\u2019t work." His accent is British Midlands. His outfit is L.L. Bean. His home-brewed coffee is Starbucks, but "better than they make it at the store."He spent several years programming at Coopers and Lybrand back when it was one of the Big Eight. Then he worked for eight years as an independent consultant for large software projects. "Programming, requirements analysis, I\u2019ll even do project management if you force me to," says Fowler. He is also one of the 17 founding fathers of The Agile Alliance, a group dedicated to revolutionizing software development. The group members met for the first time in February, in ski country on Utah\u2019s Wasatch Range. They knew something was wrong with the way software was being built, and they wanted to do something about it.Coming in, no one agreed on precisely what would work. Kent Beck, a project manager for hire, touted Extreme Programming, a system that breaks a project into tiny steps. (See "Variations on the Agile Theme".) You\u2019re not allowed to go on to the next step until the first is proven to work. Jeff Sutherland, a methodologist and consultant, was wont to use a method he invented called Scrum, in which teams of programmers went over each other\u2019s work every day. Jim Highsmith, yet another project management consultant, had published papers on Adaptive Software Development, which emphasized close collaboration between business and IT. And there were several other theories and methodologies.Surprisingly, the 17 developers in fact were able to weave the common threads of their philosophies into the fabric of The Agile Alliance.Agile means what it sounds like: fast and efficient. Small and nimble. Less money, fewer features, shorter projects. The Alliance framed a manifesto and posted it on its homepage (www.agilealliance.org).How to Be AgileSlash the budget. Small budgets force developers to focus on the essential. Small budgets also make it easier to kill failing projects. For example, imagine a project that has already cost $20 million. Imagine it\u2019s going down the tubes. With that much skin in the game, it\u2019s tempting for the CIO to invest another $10 million in an attempt to rescue it rather than take a huge loss. All too often, he ends up with a bigger one.Jim Johnson, chairman of The Standish Group, says he forced the CIO of one Fortune 500 company to set a $100,000 ceiling on all software projects. No exceptions without approval from the CIO and CEO. Johnson claims the company\u2019s project success rate went from 0 percent to 50 percent. If it doesn\u2019t work, kill it. Bring marketing, program and project management, and IT executives together at the beginning of a project and as it progresses to evaluate every piece of code in development. Is it doing what the business wants? Is it working? Any code that isn\u2019t should be mercy killed. This is called triage, and it\u2019s "the perfect place to kill a software project," says Pat Morgan, senior program manager at Compaq\u2019s Enterprise Storage Group. He holds monthly triage sessions and says they can be brutal. "At one [meeting], engineering talked about a cool process they were working on to transfer data between GUIs. No one in the room needed it. We killed it right there. In our environment, you can burn a couple of million dollars in a month only to realize what you\u2019re doing isn\u2019t useful." Keep requirements to a minimum. Don\u2019t start with everything you want the software to do; start only with what it absolutely must do. And don\u2019t worry about writing all your needs down, because requirements change.Every software project traditionally starts with a requirements document that will often have hundreds or thousands of elements. But The Standish Group estimates that only 7 percent of the features of any given application are actually needed. And a major reason for software failure is feature overload?when a programmer adds a feature that interferes with an essential process. Fixing that disconnect in turn creates another disconnect, and so it goes.Tom DePauw, manager of IT at Cater-pillar Financial Services in Nashville, Tenn., is using Agile Development to build Cat FinancExpress, a massive Web-based system that integrates three older software systems used for helping customers finance heavy equipment purchases."When the project started, Caterpillar Inc. [the parent company] wanted to see the book," DePauw says, referring to the requirements document. "I held up this single sheet of paper with a diagram on it and said, \u2019This is it.\u2019" Build on success, not hope. As often as once a week, and not less than once a month, complete a piece of software. Then have your business deciders test and approve it.That doesn\u2019t mean the software is shipped or deployed, but it must work and it must be bug-free. This is the Agile concept\u2019s most radical departure from traditional development. In some software projects, no one shows working code for years.Ken Moskowitz, CIO of New York City-based Standard & Poor\u2019s, dictates weekly builds of his company\u2019s software. Every week, the pieces being worked on are compiled and tested. "We\u2019ve had a great deal of success with it," Moskowitz says of this rigid weekly schedule. "I don\u2019t want things thrown across the transom. I don\u2019t want \u2019Here, I think I need this. Go and build it, and we\u2019ll see if it was what I really needed in the first place.\u2019"Keep your development teams small. The fewer developers the better. Developers should team code.Proponents of Agile Development swear that team coding is more efficient and produces stronger code than solo efforts. But even Fowler admits that this is one of the harder tenets of Agile Development to accept. Who is teamed and why? How do you budget their time? Expect a learning curve.Caterpillar\u2019s Agile project (in progress for two years with a core team of 15 developers) encompasses 200 distinct financial screens and contains 1.5 million lines of code. Assign non-IT executives to software projects. Non-IT execs should coordinate with the technical project manager, test iterations to make sure they\u2019re meeting user needs, and act as liaison between executives and IT.With business involved full time, "It\u2019s hard for us to say after three months of iterations that we didn\u2019t really know how it was going," says Chris Colleran, CTO of a Salt Lake City market research outfit called SpreeRide. Colleran is using Agile Devel- opment to set up his company\u2019s website and some back-end applications and has business executives full time on the project."Emotionally it\u2019s hard to commit businesspeople to the development process, but it\u2019s only counterintuitive because of the way it has always been," he says. "If you think about it, the perfect developer would be half a businessperson and half a programmer."No More Software Development Failures, PleaseFowler has joined ThoughtWorks, a Chicago-based software developer that uses Agile techniques almost exclusively. ThoughtWorks has worked on the Caterpillar project, and it employed iterative development on a retail application for The Gap. For Cisco, it integrated sales software with financial tracking systems. "The way software is developed is fundamentally flawed," says ThoughtWorks CEO Roy Singham. "Think of it as climbing Mount Everest. Do you set a plan and say, \u2019On day seven we\u2019ll be here and the weather will be this and I\u2019ll need my ice pick\u2019? You can\u2019t do that."CEOs have irrational expectations for what [software] can do, and CIOs don\u2019t have the gumption to tell them the truth," Singham adds. "Agile is a much more honest approach. I believe it\u2019s the next wave."So does Kolence, who thinks Agile Development is the best hope yet to render his 1967 paper irrelevant. For Kolence, it\u2019s been a long, frustrating struggle. "At North American Aviation in the early \u201960s, I was responsible for programming commercial applications at one of the airports," he recalls. "We never got anything done on time."Just last year, Kolence worked with an aerospace company that couldn\u2019t write a requirements document, he says. It got so bad, he told off the head of the project and warned him that they\u2019d end up trashing the software. Which is exactly what they\u2019re doing now."It\u2019s frustrating, especially when it\u2019s all ad hoc stuff," Kolence says. "There are times I\u2019m just so unhappy with the whole field. You can hear the strain in my voice. This is why I\u2019m semiretired. I\u2019m working at a local hardware store now. It\u2019s fun. It\u2019s fun to deal with people who are fixing stuff. It\u2019s fun to say to them, \u2019This is how you do plumbing.\u2019"