7 things CIOs should know about agile development

Though it may surprise cynical IT managers to learn this, developers care passionately about the software they write.

Managers want top-notch, bug-free applications delivered on time, which meet or exceed user expectations—and the people who build that software have the very same intention.

How managers and developers achieve those goals, however, can be far different. The “old” school of software development followed a path in which software specifications were defined explicitly in the form of an inviolate requirements document. Then managers told the development staff to write that application…and come back when finished.

The result, too often, was a great answer to the wrong question: software that’s late, that does interesting things but doesn’t solve the user’s need, and—well, you’re probably too familiar with the results.

Over the last decade or two, software engineers have invested a huge amount of energy in finding better methodology for creating great software that users love. Agile development embraces several processes, such as test-driven development, and extreme programming (XP). There are quite a few more processes, each with its adherents and evangelists. But they all consider themselves agile.

They have quietly been taking over your development department, too. According to recent research from Evans Data (registration required), more than half of North American developers are using agile methods today. But IT managers may not be on board.

Agile has crossed the chasm, and “the majority of organizations are doing agile, and agile works better in practice,” says Scott Ambler, author of several books (including Agile Database Techniques: Effective Strategies for the Agile Software Developer) and practice leader for agile development in IBM’s Methods Group.

If you aren’t a programmer, or if it’s been so long since you coded that you get nostalgic for JCL, it behooves you to listen up. I asked online of more than 50 developers, “If you could get the boss to understand one thing, just one thing, related to agile development…what would it be? Why that?” Their summarized answers will help you prioritize the essentials to incorporate these development practices into your software projects.

Steven Smith, CIO of Lake Quincy Media and also a software developer, wraps up most developers’ opinions succinctly: “I wish management understood that the quality value provided by agile technologies, and most especially from well-tested code, pays big dividends in terms of reduced QA expense and faster time to market for any significant-sized project. We have some projects in-house that were developed in an agile fashion, and others that were not—we dread touching the ones that were not.”

1. Agile creates better software

Agile methodologies aren’t something that your development departments adopt because it’s a cool way to write code. They see the major benefits in business terms, just as managers do.

In the experience of Kelly Anderson, C# developer and technical team lead at Sonic Innovations, the agile approach is less expensive. “If you count the cost of bugs that make it too far downstream in the process, there is simply no way to financially justify any approach except some kind of agile approach,” he says. “Poor quality costs money. Lots of money. Agile leads to higher quality, enough higher that it is cheaper.”

Plus, Ambler adds, it’s easier to govern agile projects than it is to govern traditional projects.

“Agile development can reduce the cost of late changes, making it easier for IT organizations to respond as stakeholders’ needs evolve,” says Cem Kaner, author of Lessons Learned in Software Testing and Testing Computer Software.

Kaner, who is professor of software engineering at the Florida Institute of Technology, adds: “The most important thing the executive can do is keep asking the critical question, ‘How does this practice (paired programming, test-first programming, whatever the group is proposing this week) make us more able to be more responsive later?'” If those implementing the process can’t answer this question convincingly, Kaner says, it’s a big red flag.

Agile’s business value comes from organizations’ ability to focus on which projects to do, what features to include in those projects, and which processes to use, according to Kent McDonald, business systems coach at consulting firm Knowledge Bridge Partners. “Value is provided by only doing the important things that directly contribute to the sustainable competitive advantage of the organization, and value is detracted when activities are performed that do not contribute to that sustainable competitive advantage,” he says. “Organizations can adopt a focus on value without necessarily adopting agile development practices, but it is a cornerstone of those approaches.”

“Enterprise agility means that the enterprise can respond to business needs faster—and that it can maintain this,” adds Alan Shalloway, CEO of Net Objectives. “It means that product managers select functionality that will make the most sense to the business and deliver in stages so resources can be reallocated as necessary to provide the greatest value.”

The business benefit is a key message for CIOs and IT managers. But it’s not the issue about which developers feel most passionate.

2. The agile mindset is more than processes

Why is the agile approach such a big deal to developers? They see it as a bigger change than how software is developed.

Agile is a way of working and of thinking about work, says Mike Sutton, who runs agile consultancy Wizewerx. “It challenges entrenched thinking. It’s painful because it forces organizations and individuals to confront waste, inefficiency and shortcomings (personal, professional and organizationally). It covers every part of the organization: engineering, stakeholder management (regular quality feedback and refactoring), team interactions (geeky developers must now talk to each other and the rest of the world—heaven help us) and customer satisfaction.”

You can mess with the methodology to some degree, but if you don’t adopt the mind-set wholeheartedly, assert developers, don’t bother. “I’d hope my CIO understood that half-agile is very, very dangerous,” says developer John Overbaugh. “If the organization wants to test out agile, we need to pick a project or two to be agile. What we don’t want to do is make each project ‘somewhat’ agile.” Doing so, he feels, dooms all to failure. Part of what an agile developer rejects is cultural impediments and attitudes, including a management demand for command and control, or customers and business analysts expecting to throw the requirements over the wall and ending up with an “agile” result.

That’s why support from the top is critical to making agile development succeed in an organization.

[ Related: 4 critical roles for agile success ]

Often, its adoption is from the bottom up—developers seeing agile as a solution, and then selling it to senior management…or trying to. CIOs need to adopt the agile attitude, these folks say, or it won’t be accepted successfully throughout the company.

Process-oriented managers who are used to the waterfall model may have a hard time adjusting to the different philosophy, and that can cause friction.

In the waterfall model, software development is seen as flowing steadily downwards (like a waterfall) through different stand-alone phases: requirements analysis, design, implementation, testing, integration and maintenance.

Agile developer Tim Ottinger suggests that IT managers begin by looking at agile values, to learn whether they can really buy into the methodology. “Agile requires a certain release of control and change of values,” Ottinger says. “If they’ve enshrined long hours and centralized control, or if they’re suffering through relational problems (labor/management style), or if they want larger commitments done faster instead of more measured, regular success…well, you see. There is a reason that some companies have a lot of trouble transitioning.”

So what are those values? Jeff Tucker, software engineer at Milliman Care Guidelines, explains, “Agile is basically the idea that requirements change because people change. You develop software so that you get rapid feedback and can accommodate change rapidly and easily…. As long as you’re doing something that realizes that benefit, you’re agile. A lot of executives either don’t understand that and therefore fail to see the benefit, or they’re extremely dogmatic about it. And either of those situations is a disaster that’s either happening or waiting to happen.”

Sutton adds, “If you look at the U.K.’s only real huge scale agile success, [global service provider] BT, it took the CIO, Al-Noor Ramji, to provide real executive leadership and say, ‘You have to do agile.’ That is the best support for the real innovators in development teams to roll out process agility.” After BT nailed the process (Scrum, XP and a few others thrown in) and they agreed on a language for now (Java/JavaEE), says Sutton, their ongoing task became challenging people and entrenched cultures.

3. Agile changes more than development workflow

Part of an organization’s tackling the agile development process is accepting that it will—and should—change company culture as well as process.

The important changes include pervasive use of feedback and increased collaboration—factors that force business and IT colleagues to be truthful with each other and bring visibility into problems an organization may have to address.

Consultant George Dinwiddie from iDIA Computing suggests using a burn-down chart to track project progress. Doing so, he says, shows how progress is converging (or not) on the goal line and lets a manager know when to take action. “That action might be to move the goal line, or to make changes in the way the software is being produced to try to speed it up.”

When a developer asks the business customer about a feature, he gets feedback on the desired goal; the customer can see from the incrementally growing software how well it works. Says Dinwiddie, “The feedback should be generated with as little delay as possible. Reducing the time between when a decision is made and when it is validated will pay off in reduced waste.”

Another effect is increased collaboration…as long as management recognizes its side effects. “Everyone in the organization should be collaborating, with the organizational end goals in mind, instead of just minding their own business and doing their assigned jobs in isolation,” says Steven Gordon, an independent agile software development coach.

Note: That collaboration brings to the surface a lot of issues, problems and obstructions in the organization. Do not kill the messenger, Gordon urges; agile is not creating these issues, only making them more apparent. CIOs have to prioritize and address the problems. “If the organization appears to be ignoring these issues after they are surfaced, people will come to believe that the agile principles make no difference and will go back to just minding their own business and doing their jobs in isolation,” Gordon says.

But in the widest sense, a long-term effect of agile development may be truthfulness. “I won’t mince words: If you can’t be truthful, you can’t do agile successfully,” says Mishkin Berteig, president of Berteig Consulting. “Truthfulness is the foundation of the practices, it is the foundation of even the principles. If you look at the agile manifesto, you can see that the idea of truthfulness is at the foundation in communication, in human interactions, in construction, and in customer-vendor relationships.”

Berteig is often surprised how often this concept is a revelation to executives. “Most of our organizational cultures in the corporate world are based on distrust and a decided lack of truthfulness. To change to a culture of truthfulness is a deep, long and difficult change—just like any other culture change!”

Yet, he asks, without a culture of truthfulness how can one learn from the inevitable mistakes? How can an organization create mutually beneficial relationships with clients? How can you be certain that at the end of each iteration the team is delivering what it really says it’s delivering, or be certain that the stakeholders really want what they say they want? Berteig adds, “Traditional bureaucratic processes and procedures are often a result of a lack of truthfulness and an attempt to make up for that deficit.”

4. Agile doesn’t mean ‘chaos is good’

Managers who are comfortable with one-step-at-a-time methodologies may be uncomfortable with the iterative nature of agile development. But agile development doesn’t imply disorganization.

Isaac Sacolick, vice president of technology at BusinessWeek and previously founder of TripConnect, says that while agile gives sponsors the ability to react to market conditions and reprioritize requirements, “Agile does not preclude long-term planning, architecture, standards or project management. Enterprises have to think through how and where to apply agile development.”

The how and where matters. Few developers believe that agile is appropriate for every project and for every team—and they hope the CIO understands when to set it aside. Marshall Presnell, development architect for Village Voice Media, says, “The agile model of development is optimally suited to rapid and chaotic requirements changes.” And Gary Rowe, director of Froot, an agile development consultancy, recommends agile for projects with unclear or shifting client requirements.

Where’s it the wrong choice? John Miano, owner of Colosseum Builders, says it’s not suitable for projects that will have a long lifetime and those where complete reliability is required.

Organizations that adopt agile principles don’t change the basics of defining problems. “Agile development doesn’t mean that you don’t do requirements analysis and design,” says Mike Emeigh, QA Manager at Deltacom. Good agile teams do requirements analysis and design, he says. “They just don’t waste a lot of time committing it to reams of paper [or] have a whole bunch of people who can’t possibly grasp all of [the issues] try to study and understand it before they commit it to working software.”

But the iterative nature of agile development—in which developers and users work together to design and cocreate the application—sure doesn’t sound like it’s easy for a manager to track. CIOs want to know the status of every project in company, but agile implies that requirements and schedules are always changing. Adherents insist that’s a wrong assumption.

Responds Christopher McMahon, QA tester at SocialText, the enterprise wiki company, “We estimate the work for each iteration. Because our estimations get better over time, we know from iteration to iteration exactly what we can deliver, and we can project that into the future with relative ease. Because we track stories in the backlog on burndown charts and use other Big Visible Charts to broadcast our status, that status is easy to check. And because we have Iteration Managers whose job it is to keep the work moving, we have an expert available on each project to answer any detailed questions.”

According to Ben Weatherall, configuration manager at pharmacy technology provider PDX, the workflow involves more than just the programming staff. “Just because development is practicing agile methods does not mean that configuration management should be ignored,” he says.

Agile development includes version control on code and requirements; defect, issue and enhancement tracking, and some way to relate their status to the code changed to resolve them; and a way to capture, track and index communications with the customer to indicate changes to the requirements, success or failure of demos and tests, and so on. Adds Weatherall, “The tools used to facilitate CM in an agile environment need to be ‘tuned’ to that environment just like the development tools do.”

5. Agile’s benefits are worth the wait

As with any process change, the effects won’t be seen immediately. It takes time for the process to take hold.

James Kricfalusi, service delivery executive at TEKSystems, says, “Sponsors and executives equate agile with immediate results. This is not realistic. Trust the team, relax and let them do their thing. Lower your expectations in the beginning because most of the initial advances cannot be in the functionality of the first release.”

According to Kricfalusi, in the early work on an agile project, initial iterations are an investment in frameworks, new ideas, process definitions and understanding of the requirements. The initial release may miss the mark, but the team will have developed a cheap and effective means of providing effective feedback to improve later versions. Judge the results relative to the time frame of the existing processes, Kricfalusi advises. “If your prior project time frames were 6 months to a year, it will take about that long before you can have the proper perspective to evaluate your progress.”

Don’t get lost in those initial iterations, either. Remember: They aren’t intended to be the final version. Ivan Brusic, lead Web developer at Mansueto Ventures, says, “Sometimes a prototype is just that: a prototype. Early project iterations might get thrown away, but that tends to happen only when the initial requirements were not understood.”

Sutton urges CIOs to tell everyone involved that you will support their efforts to deliver working software in a month, every month. “Back them up 100% like you promised—on training, wacky new ways of working—whatever they come up with. Try it for a month and see what you get, then do the whole thing again for the next month until you get the phenomenal results!”

For the changes to take place, says Pollyanna Pixton, president of Evolutionary Systems, stop micromanaging. “Recognize that agile practitioners will deliver business value in increments. Leaders should stop placing importance on ‘old school’ measurements such as schedule and costs, because in rapidly changing markets focus should be on value creation.”

Agile tools like burndown charts will enable managers to see value creation and value velocity, so leaders can make the go/no-go decisions early in the projects, Pixton says. And, in so doing, avoid building any of the 64% of features rarely or never used.

6. Agile isn’t a silver bullet

While agile can offer businesses a lot of improvement, it won’t solve every development problem.

“Bad programmers/communicators can still fail with agile,” points out developer Kelly Anderson. “Give competent and invested people agile, and you’ll have happier people doing better work more rapidly. Give incompetent, unempowered people agile (or even worse, just the word agile), and you’ll still fail,” he says.

Neil Roodyn, author of eXtreme .NET: Introducing eXtreme Programming to .NET Developers, says that agile will not convert a bad development team into a good one. “Agile is not a religion, a methodology or a solution to allow mediocre software developers (which is most developers, by the nature of the term) deliver awesome software.”

Ed Cowsar, CEO and managing principal at Inside Methods, says, “There is no middle ground in agile software development—radical success or miserable failure are your only options.”

Excellence requires care and attention to detail, as in any endeavor. “To successfully adopt agile development, teams need self-discipline, a rigorous approach to their work, management support and time to learn. Good managers can supply the latter two. A good coach or technical lead can help the team achieve the first two,” says James Shore, an independent consultant and author of The Art of Agile Development. “Excellence is hard; people like to take the easy way out. But the big gains people associate with agile development come from excellence, not mediocrity.”

Don’t adopt agile because it’s popular, says Shore. “Don’t go out, buy a bunch of ‘agile’ project management tools and certifications, and declare victory. Have your conscientious and thoughtful leads take the time to truly delve into the ideas and philosophy of agile development, including why those tools and certifications aren’t central to the agile approach.”

7. Success depends on people

Just as agile may not be suited to every type of development project, it may not be the right option for every team, peoplewise.

Miano, the owner of Colosseum Builders, says, “Agile-type development also tends to be very personality dependent. If you put 10 people together in a room, they all had better be compatible…. The people working in such an environment need to be able to ignore the personal quirks of the others.”

Others believe that agile can bring teams together. Steve Reiser, a senior software engineer, says that a team’s ability to respond quickly, with frequent periodic successes, makes team members empowered and happy. “When a project fails, the team fails. When the team fails, morale suffers,” he says. “Agile keeps work levels high, success rates high, adaptability high, which all contribute to keeping morale high.”

Is it a lot of work to make agile work for your team? Certainly. It requires a major change in the way that software is developed, and in the way that teams— including the user community—design applications. But, insist the people who use agile methodologies, it’s worth it.

More on agile development

• Agile project management: 12 key principles, 4 big hurdles
• What is SAFe? The Scaled Agile Framework explained
• Agile vs. waterfall: Project methodologies compared
• 16 tips for a smooth switch to agile project management
• Top 11 agile tools for project management