How Agile Development Can Lead to Better Results and Technology-Business Alignment

Farm Credit Services of America doesn’t sound like an organization that courts controversy. The cooperative association makes loans to more than 66,000 Midwest farmers and cattle ranchers so that they can buy cows and pigs and tractors and backhoes. Its main reason for existence—providing $11 billion of operating capital and real estate financing to those who feed America—is as homey as the images of corn fields, gently rolling green pastures and rugged, resolute farmers that adorn its marketing materials.

It’s also based in Omaha, Neb., known more for steaks than as an avant-garde laboratory for one of IT’s most hotly debated development methodologies: agile programming.

But agile is exactly what Farm Credit Services has embraced, whole hog.

HOW AGILE ARE YOU?

Are your developers agile? See this 10-question list to find out.

  The Agile Advantage

Agile programming means different things to different people, but at the core of all agile development methodologies are these principles: Business stakeholders are colocated with small, autonomous development teams; the teams rely less on up-front requirements and documentation than on face-to-face conversations; those conversations provide a continuous dialogue for software design, testing and refocusing. The constant refocusing, its advocates say, leads to more timely and useful business tools. (For a tutorial on agile programming, see “ABC: An Introduction to Agile Programming”.)

Agile’s ascendancy is in direct response to IT’s dolorous history of software project failure, cost overruns and the concomitant business dissatisfaction with traditional IT design and development—the waterfall methodology—in which development slowly cascades through a series of steps including requirements analysis, design, implementation, testing, integration and maintenance. But for a variety of reasons, not everyone has warmed to agile. In fact, just 17 percent of North American and European enterprises use agile development processes, according to Forrester Research’s “Enterprise Agile Adoption in 2006” survey.

Farm Credit Services welcomed agile programming because the waterfall method had been failing the organization, as it has many others. “We got requirements and would build [the applications], and nobody was happy at the end,” says Farm Credit Services CIO Dave Martin. One particular project, which was a migration from a mainframe-based customer application-processing system to a Web-based version called PinPoint, involved more than 200 pages of requirements and, by the end of 2004, had taken nearly three years to complete. In the interim, the requirements and business needs had changed, and most of the members of the original business team were gone. The resulting bug-filled system was shelved not long after its shaky debut.

And that’s not unusual. According to the 2006 “State of Agile Development” survey by The Agile Alliance and Version­One, respondents said only 29 percent of traditional projects were “somewhat successful” or “very successful.” (Conversely, respondents said 81 percent of their agile projects achieved that level of success.)

The Standish Group, which famously compiles its Chaos data on software project failures, reported in its 2006 research that just 16 percent of waterfall projects succeeded as opposed to 41 percent of agile projects. Standish Group Chairman Jim Johnson, who has been studying project failures for years, says it “boggles” his mind why companies still resist agile development. “To say that companies or CIOs are reluctant to embrace agile is like saying they wouldn’t take aspirin for a headache,” he says. “And they’re not only not taking the aspirin, they’re banging their heads against the wall and wondering why it hurts.”

Going Agile

Martin decided to “shake things up” two years ago. After some gentle nudging from Lou Thomas and Beth Schmidt, his directors of applications development, the team adopted Scrum, which has two-week iterations (called sprints), daily meetings, and frequent iteration reviews and testing. “Our premise is to have potentially shippable product every two weeks,” says Thomas.

At Farm Credit, there are six development teams composed of a business analyst, project leader, lead developer, two or three developers, a database engineer, one to three business owner participants and a QA engineer. In place of reams of requirements, development teams write “user stories” as the project progresses, detailing business functionality enhancements and technology, successes and challenges. “User stories are the main mechanism to convey the business needs,” Martin says. Farm Credit’s waterfall projects used to average 100 or so defects per rollout; agile ones now average zero to two.

“We’ve rolled out five key products with phenomenal results,” Martin says. “In each case our business owners are ecstatic with the end result.”

Agile, Agile Everywhere? Well, No.

CIOs of companies large and small say they have either abandoned waterfall methodologies or are gradually phasing them out. “I don’t do the ‘I’ll deliver everything to you in 18 months,’” says Raymond Dury, CIO of Fifth Third Bancorp in Cincinnati, who has adopted an agile method called Extreme Programming. “That’s long gone.”

But waterfall processes are not, in fact, long gone. In addition to the aforementioned Forrester survey (just 17 percent using agile), a March 2006 survey that polled readers of Software Development magazine and Dr. Dobb’s Journal found that 60 percent were not using any agile methodologies in their organizations at all.

Perhaps not coincidentally, CIOs are actively looking for project management assistance. Almost three-quarters of CIO readers are either “extremely interested” or “very interested” in finding out how to improve their project management discipline, according to our latest survey.

All this leads to one obvious question: If agile development is so darn good, then why hasn’t it been universally adopted?

The Trouble With Agile

The ceremonies of software development are deeply ingrained in IT. With traditional waterfall processes, the business throws its requirements over the wall to developers who hole up and start coding as they see fit. An 18-month target date can seem like decades away. Lost afternoons are no big deal. Who cares what the “lusers” (coders’ derogatory term for users) really want?

“A lot of people on the IT side thought [agile] was the flavor of the month,” Martin recalls. “Some just said, ‘I’m not going to do it.’” (Those programmers, Martin says, have been churned in agile’s wake.)

Opposition to agile methods also can come from enterprise architects, project managers and quality assurance staffers, says Carey Schwaber, a senior analyst for application development at Forrester Research. Enterprise architects worry that there’s not enough up-front design with agile, and the consequence is spaghetti code. Agile teams are self-managing, and the project leader’s role shifts dramatically—from ordering around to facilitating. And since QA testing happens throughout the process, and not just at the end, there’s usually resistance from the testing folk.

Misapprehensions about agile still run rampant in IT organizations. Eugene Nizker, a former financial services CIO and current consultant, ticks off the most infamous ones: Agile teams do not plan. Agile teams skip design. Agile teams do not test. Agile means no documentation.

In addition, executives can feel left out of the daily scrums and sprints of agile life, engendering insecurity at top levels. All this has hindered agile’s acceptance, says John Scumniotales, one of the creators of Scrum. “It’s easy to talk about the value of building software this way, but if I’m betting my enterprise on this project, senior management needs some controls and visibility into the process,” he says, citing the need for an agile-specific tool that functions like a Gantt chart, which visually illustrates project progress. “That’s where we need to get to,” he says.

But given the epic floods of waterfall failures, CIOs owe it to themselves and their organizations to get agile. “The world just doesn’t hold still and wait for 18 months anymore,” Fifth Third Bancorp’s Dury says.

Case Study: Quick to Market

Dury hasn’t abandoned traditional development processes at his company, which has almost $100 billion in assets. Some development projects, he says, such as those that affect base infrastructure or intensive SOA projects, just “can’t handle the type of intense change” that comes with agile methodologies. But he’s excited about his team’s adoption of Extreme Programming.

The head of the bank’s retail product line told Dury about a new product the bank wanted to sell, an investment tool (which would later be called Stock Power CD) that would offer customers stock market–based earnings potential on their certificates of deposit. In truth, the bank was playing catch-up. “We were probably missing the market [for this type of product],” Dury says, “and we wanted to be in the market.” So speed was key.

Using Extreme Programming tactics, his team talked through solutions that would meet the business needs quickly and be technically feasible. More conversations ensued with retail banking (lots of face-to-face time, little up-front documentation). Two development teams, working in concert with each other and the consumer banking and retail line departments, formed within IT: the investment adviser team and the CD development team. The teams worked closely with the bank’s affiliates (which would eventually sell Stock Power CD) and investment advisers to satisfy legal and financial requirements.

Extreme Programming—daily huddles and continuous business input, as well as recurrent testing—“created the urgency to provide this service to the customers ASAP,” Dury says. Just four weeks after the initial conversation, several of Fifth Third’s affiliates started offering the Stock Power CD product. “Normally a new deposit product introduction would take about 90 to 120 days,” Dury notes, adding that the new product created “significant additional deposits.”

Dury says he couldn’t imagine doing this project any other way. Along with the faster time to market, it also enabled “a closer working relationship with the business side” and enhanced the business’s confidence “in our ability to deliver and meet expectations.”

The Alignment Add

As Martin’s and Dury’s stories show, agile project methods demand continuous dialogue between the business and IT. Colocating the business and IT team members engenders a kinship that’s usually absent in traditional software development.

“They’re on the ride with you,” says Ajay Waghray, Verizon Wireless’s CIO of the Midwest U.S. area. “If something is not going right, they’re partnering with you right there.”

CIOs who have adopted agile methods say that changing their relationships with business stakeholders can have some surprising results. Waghray recalls recent conversations with key operations and marketing executives in which he used a more agile approach. “I would tell them on the call, ‘I don’t want any requirements. What do you want? Just tell me, and we’ll talk about it,’” he says. The executives asked if they should write down what they wanted. “And I would say, ‘No, no, no. Don’t give me requirements.’” The approach, he says, “was foreign to people.”

Verizon Wireless’s VZ Navigator product, which provides users with turn-by-turn, voice-supported directions on their mobile devices and has become one of the company’s most popular mobile services, was heavily agile-influenced. Waghray’s team consisted of stakeholders from IT, sales, customer care, marketing, training and finance. The team’s main focus, Waghray says, was to develop the “simplest complete” solution right away, and then build upon that to make better solutions. “The reason it was successful was that the [agile] methodology encouraged constant communication and alignment across all stakeholders, leading to parallel development of not only the IT solution but also of training documentation, test cases, external and internal interface validation and development, and user communication to get them ready for launch,” he says.

To many of the businesspeople on the team, the results were shocking. They told Waghray that this project should have taken three times as long as it did, which was roughly eight weeks from start to finish. According to Waghray, the VZ Navigator project “made a significant and immediate impact on the bottom line and it has now enabled an organizational capability to do this with similar products.” It is “a culture and timing change,” Waghray says of the process, “but I have never gotten more accolades from a business project.”

Managing the Agile Change

Since agile teams are designed to be autonomous bands of rapid change agents making critical business decisions on the fly, the once-immutable laws of project management shift dramatically. Because waterfall’s typical 18-to-24-month development times are drastically reduced to two-week windows, every day of development and every conversation become critical. In Dury’s IT shop, mornings start with a team huddle and a breakdown of what everyone is responsible for completing that day. “We don’t need to talk about what gets done three weeks from now—this is what we want to do today,” Dury says. “We don’t want to lose days.”

It’s incumbent on CIOs to set the new expectations—teamwork, openness, collaboration—for everyone in their agile group. “There’s an expectation of collaboration. You can’t just go work in your own little world,” says Farm Credit Services’ Martin. “There’s also a new visibility into the work and, in some cases, the nonwork.”

Those team members who resist, however, will more than likely find themselves out of a job. (An old Persian saying seems appropriate for CIOs to remember: “The dogs bark, the caravan passes on.”) Scott Ambler, an agile expert who works as practice leader for agile development at IBM, suspects that many developers, requirements analysts, and data and testing staffers are worried about agile’s career-changing consequences. “They have likely worked on a slew of failures, and they often feel powerless to change things. Worse yet, they have the threat of outsourcing hanging over their heads,” Ambler says.

The CIO’s job is to manage the anxiety and ensure that his team focuses on producing high-grade software in a timely and cost-efficient manner. “In the agile community, the only measure of progress on a software development project is the delivery of working software,” Ambler says. “The traditional development community has lost sight of that.”

Why Are So many CIOs Cool to Agile Software Development?

Skepticism is the word most often used when CIOs and analysts are asked why so many CIOs have been cool toward agile software development. They hear about it at a conference or in an article and it sounds good, but the 2006 “State of Agile” survey reports that only 5 percent of respondents cited the CIO as being the initial champion of agile development processes. Surprisingly, 11 percent said their president or CEO was driving agile acceptance.

Without the CIO’s backing, as well as support from influential business stakeholders, agile’s software development and project management time and cost efficiencies cannot be realized. “When companies don’t have a VP-level or CIO-type executive driving adoption of agile, they will run into so many obstacles,” says Forrester’s Schwaber. At Farm Credit Services, Martin’s agile move has grabbed the attention and support of his CEO. There are no obstacles (besides keeping up with project demand), he says.

The agile mentality has even proliferated throughout the rest of the organization. Martin knew he’d made the right move when he’d heard that business teams were holding daily stand-up meetings—mirroring IT’s practices. “We’ve even got business units asking, How can we be more agile?” he says.

Martin’s agile journey has come full circle. He’s gone from being a CIO who didn’t know anything about it or anyone doing agile, to an agile evangelist, spreading gospel at conferences, trade shows and CIO breakfast meetings. “I couldn’t fathom going back to a waterfall methodology,” he says.

Senior Writer Thomas Wailgum can be reached at twailgum@cio.com.