Brownfield Development: An Agile Approach to a Waterfall Problem

The "Brownfield" application development lifecycle differs from that of the traditional waterfall method employed on large-scale developments and instead utilizes a more agile approach, write the authors of Eating the IT Elephant, in this sample book chapter.

Chapter 8: Brownfield Development

"The process of scientific discovery is, in effect, a continual flight from wonder." —Albert Einstein

Chapter Contents

  • Agile Waterfalls
  • The Brownfield Development Approach

The previous chapter discussed the methods and techniques that form the foundations of Brownfield. We investigated each of these and compared the differences and similarities. Brownfield was shown to have evolved from these methods.

This chapter shows how the Brownfield method is used within the project lifecycle. As in the previous chapter, the lifecycle approach has again evolved as a composite of a number of existing lifecycle approaches.

Excerpt from Eating the IT Elephant: Moving from Greenfield Development to Brownfield

book cover image

Agile Waterfalls

Large-scale development has traditionally been based around well-organized phases. Each phase is completed before moving on to the next phase and is defined through work breakdown structures, work product definitions, and quality criteria. When viewed on a plan, these phases form steps down the page; this picture has led to the term "waterfall development." The water moves in only one direction, filling each phase before moving on to the next phase.

The more complex projects get, the more rigorous the methods become—up to and including extensive use of static testing and formal systems engineering techniques. These techniques are used to form quality assurance checkpoints at key points along the waterfall.

Static testing consists of techniques such as inspections and design walkthroughs to test the system's requirements, design, and implementation before the system can be executed.

The checkpoints check progress and create rigorous, strong baselines. The baselines are made up of a single set of unambiguous, formally signed-off documents. Figure 8.1 shows these project phases together with the quality assurance checkpoints used to enforce the rigor. The example shown uses the baselines defined in Carnegie Mellon's Software Engineering Institute's Capability Maturity Model Integration (CMMI).

elephantbook08fig01-small.jpg
Figure 8.1: The major phases of the waterfall lifecycle are punctuated by formal gates that allow baselines (often commercial ones) to be defined.

On very large projects, many people have tried to shortcut this process (deliberately or through ineptitude, or both), and many have come unstuck.

As Barry Boehm and Richard Turner point out in their book, Balancing Agility and Discipline: A Guide for the Perplexed, five factors determine whether waterfall or agile methods will prevail on any particular project. For the purposes of this book, we have rephrased them, but we have stayed true to the spirit of their findings (see Table 8.1).

Table 8.1 Comparison of Agile and Waterfall Methods

Factor

Agile Characteristics

Waterfall Characteristics

Size

Optimal for small projects and teams; reliance on tacit knowledge

Tailored for large projects and teams

Mission-critical projects

Untested; general lack of documentation

Long history of use in such implementations

Stability and complexity of existing

Continuous refactoring used; suitable for dynamic and simple environments (typically Greenfield)

Structured baselines used; suitable for more static and complex environments environment (typically Brownfield)

Skills

Continuous involvement of highly skilled individuals; difficult to cope with many lower skilled resources

Highly skilled individuals needed in early phases; designed to cope with many lower-skilled resources in later phases

Suitable organization culture

Chaotic; dynamic; empowered

Roles well defined; procedures in place

As you can see, agile and waterfall development each have their strengths and drawbacks. To recast the comparison, it is both possible and safe to build a paper airplane without a detailed plan. It would be foolish to spend 20 minutes writing the instructions and then spend 20 seconds building the plane. However, building a passenger airliner without detailed, upfront design would be a long and expensive process involving a lot of rework that you would otherwise have avoided. (You'd also probably face a shortage of test pilots to take the airliner on its maiden flight.) Figure 8.2 summarizes the different development techniques used in building a paper airplane and building an airliner.

elephantbook08fig02-small.jpg
Figure 8.2: The development techniques used for a paper airplane and an airliner should be quite different.

In formal waterfall methods, defects are detected as early as possible through static and then executable testing. If defects are found, changes are made to the requirements, specifications, or solution design documents. Changes can ripple forward from the first work product affected to the last. This approach reduces the overall number of defects and is far more cost-effective than not following these best practices because it reduces the number of surprises and the amount of rework.

Stringent baselines and static testing of work products improves overall quality and helps ensure that more defects are found earlier. However, this is not a cheap method of delivering systems. A high level of process and discipline is required, and the method does not take into account the complexity of the environment much more than any other Greenfield method. The knowledge and impact of those thousands of constraints is still held over to the end of the project, typically into the late testing phases when changes are expensive, even with the best possible walkthroughs and inspections.

MORE ON CIO.COM

The Rising Importance of the Enterprise Architect

10 Questions To See If Your Company's Developers Are Agile

Why Now's the Time to Tear Down Old Legacy Apps and Rebuild

Source Code Management Systems: Trends, Analysis and Best Features

In reality, even in rigorous projects, a detailed requirement fault detected late in the waterfall lifecycle generally does not result in a change to the original requirements and demand that the subsequent work products and baselines be re-created and revalidated. Generally, workarounds for such problems are found and the proper solution is postponed to the next release.

Traditional waterfall methods decompose the problem to ever-smaller Views until the View is sufficiently small, detailed, and self-contained for it to be built by an individual, often in isolation. These Views are then tested in the reverse order of the decomposition, up to the final stage of a complete acceptance test. Unfortunately, this encourages a testing of the system by Views, too. Some of those Views (such as the integration or data migration elements) are often tested late and are expensive to fix. As you learned earlier in this book, it's cheaper to fix things at the front of the process. Figure 8.3 shows the decomposition and testing of the views together with the increased cost of finding an error in that phase.

elephantbook08fig03-small.jpg
Figure 8.3: The cost of removing a defect increases exponentially as time progresses; unfortunately, the industry "best practice" test model encourages late testing of some areas of high uncertainty (such as integration).

This test model, known as the V model because of its shape, is currently the industry's best practice, but the requirements testing is performed only at the latest stage of the project lifecycle. This is the most expensive point in a project to find such errors and undoubtedly causes delays in what seemed like a project that was on schedule. Surely, a more cost-effective way of testing would be to follow the approach shown in Figure 8.4.

elephantbook08fig04-small.jpg
Figure 8.4: An iterative approach that dives deep through the testing procedures would encourage earlier testing and cheaper defect removal. The difficulty with this approach, however, is that it requires an agile approach, which could also work with highly complex, heavily constrained designs.

This alternative model preserves the traceability of the original testing model, but the testing of the corresponding baseline is performed as soon as possible. In addition, "deep dives" are repeatedly made by building some aspects of the system early and testing them against the requirements; this enables at least some of the late-stage testing to be brought forward. This approach clearly looks a lot like agile development, and it is—agile works by effectively implementing some of the requirements and then using this to capture further requirements as deltas for the current situation. This is repeated until the total requirements are captured.

This can equally be attributed to Rapid Application Development (RAD). Agile is used in this case as the current best practice of RAD-type development.

Unfortunately, the agile approach works well only for Greenfield projects or medium-size projects in simple environments; the approach is often flawed for bigger, more complex solutions or those that have a lot of constraints. The previous chapter detailed an example of a Greenfield approach destroying an essentially Brownfield project with the creation of a Web front end on a complex credit card processing legacy system. In that particular project, agile techniques were used to determine the user interface and Domain Object Model (DOM) that formed the basis of the new system. The DOM defined the business information and structures used to define the problem, and these were developed iteratively with the business. Alongside the DOM, user interface prototypes and object models were iteratively refined based on business user feedback.

However, the constraints of the legacy system were poorly understood, and the DOM that was created was inherently incompatible with its surrounding environment. The iterative approach did not allow those elements to be properly understood early enough.

Waterfall techniques are excellent at dealing with the nonfunctionals, good at dealing with functionals, and reasonably good at dealing with constraints. Agile is excellent with the functionals, poor with nonfunctionals, and very poor with constraints.

The options for a sizeable reengineering project seem to boil down to these:

  • Finding out all the requirements, including constraints, before starting, and designing the ideal system to implement them (very expensive)

  • Growing the solution and discovering all the requirements as you go, but with the effect of building a suboptimal system and the danger of substantial rework (also very expensive)

We need a technique that combines the best bits of agile and waterfall but that sorts out how to handle constraints. What has the industry achieved in this space so far?

Agility under Waterfalls, or Waterfalls with Agility?

Boehm and Turner look at good examples of hybrid waterfall/agile approaches in their book Balancing Agility and Discipline. One example takes a look at an agile project of considerable size and another considerably sized, mission-critical project with agile-like deadlines and targets that demand agile behavior.

Looking at a successful project that used agile programming techniques to deliver a system of the magnitude we are talking about, Boehm and Turner found the following to be true:

  • The effort to deliver features increased nonlinearly over time (well outside the bounds of a normal agile project). As you might expect from earlier chapters of this book, this was ascribed mostly to communications overheads.

  • Increased formality was required around the design. Architectural perspectives were required to divide work and to ensure a valid design for areas that were both complex and subject to change.

  • Tacit knowledge was not enough to define all the requirements (especially for specialized use of complex function) and did not scale sufficiently.

  • An "integration" gap (of some 12 weeks) arose between "zero defect" functionally "complete" code and actual delivery of working integrated code.

As a result, some of the key tenets of agile programming had to be diluted or lost completely from their hybrid approach. These findings strongly match our experience in using agile techniques on large projects.

Using another example in the same book, we look at a fundamentally waterfall-created, mission-critical system that applied agile techniques to accelerate its development. Boehm and Turner noted that agile development on its own was not enough to maintain control, and they had to introduce the following innovations:

  • The documentation for the system needed to be automatically generated from the architecture and code artifacts.

  • The early phases of the project were used to design and develop robust patterns that could underpin the elements developed with agility.

  • Integration and performance testing was performed early and often instead of being performed at the end.

Both of these examples are telling. The three points mentioned are also techniques we have used on Brownfield projects. Certainly, we should expect any "hybrid" method that contains the best of agile and waterfall to offer the kind of capabilities identified in the previous best practices examples. We believe that the Brownfield approach enshrines such a method that enables all three kinds of requirements (functional, nonfunctional, and constraints) to be applied iteratively and incrementally throughout a project without losing control.

1 2 3 Page
Join the discussion
Be the first to comment on this article. Our Commenting Policies