Learning Scrum: An Empirical Approach to Product Development

Perhaps the most popular agile method, scrum adds specific, required practices on top of programming and testing. This causes some people to call it prescriptive. Scrum doesn’t actually tell teams how to do the essential practices of figuring out what to build, how to build and then test, though.

The organizations that support scrum have split into the Scrum Alliance, Scrum.Org and Scrum Inc. There are other independent trainers as well. All this leaves questions about what scrum is exactly, where it came from, what problems it solves and why companies struggle to adopt it.

Here are answers to those questions.

Where Scrum Came From

In the early 1990s Ken Schwaber was working with Individual Inc. and Jeff Sutherland was at EaselCorp. Individual has a problem: Requirements change too quickly for the technical staff to get anything done. Sutherland and Schwaber independently read a 1986 Harvard Business Review article, The new new product development Game, which describes how Japanese companies developed video cassette players in integrated, multidisciplinary teams. The method was radically different than how most people at the time pictured software development: as large batches handed between technical specialties, in a process often called “the waterfall.”

The initial version of the scrum development method, from Ken Schwaber’s 1995 OOPSLA paper, “The Scrum Development Method”, is shown above. Schwaber and Sutherland joined forces, writing a paper for the 1995 OOPSLA conference called Scrum Development Process.

The Creators of Scrum

Sutherland (right) was a Vietnam fighter pilot with a Ph.D in biometrics before becoming a technology executive; he’s currently CEO of Scrum Inc. His work with Schwaber (left) resulted in the creation of scrum as a development method, the Scrum Alliance and a handful of books, including Agile Software Development With Scrum, which Schwaber co-authored with Mike Beedle.

In 2001, both Sutherland and Schwaber attended the “snowbird meeting,” hosted by Alistair Cockburn, at which the Agile Manifesto was created. (The two were initial signatories.) Sutherland’s latest book, Scrum: The Art of Doing Twice the Work In Half the Time, was released in September.

The Essence of Scrum

In Wicked Problems, Righteous Solutions, published in 1991, Peter DeGrace and Leslie Stahl call scrum an “all-at-once model.” It refers to the rugby scrum, where players work together to move the ball upfield, a few steps at a time, in a huddle. The authors mention it in passing but don’t give it detail.

Sutherland and Schwaber fill in the detail, using scrum as a tool to solve a problem at Easelcorp – where, as with Individual, requirements and priorities change so fast that developers can’t get anything done. Sutherland introduces the idea of a sprint: An uninterrupted period of time during which the team can develop and test without change. This sprint has a fixed length, starting at 90 days. After the sprint, the team will “demo” the software to customers, who react to what was built and come up with ideas for the next sprint. This slows the feedback process for the development team from the customers, typically referred to as the larger, “outer circle” of scrum. After the sprint, the team also meets to discuss what worked, what didn’t work and what to change. This is called a retrospective.

The inner circle of scrum, meanwhile, is the daily scrum, an opportunity for the team to discuss work in progress in order to accelerate team performance. The meeting is called a scrum because it resembles the rugby ‘huddle.’ The term originates in the Harvard Business Review.

Since those initial definitions, the ceremonies of scrum remain the daily scrum (or “standup”), the demo and the retrospective (or “review”), along with a planning meeting to discuss what to build.

The Scrum Guide

The three aforementioned “authorities”– the Scrum Alliance, Scrum.org and Scrum Inc. – all aim to keep the source definitions the same. Sutherland and Schwaber keep them in a single document, called the Scrum Guide, which is available free from ScrumGuides.org.

At only 16 pages, the guide balances information with information overload, trying to separate the what, such as retrospective and planning, from the how-to, such as running those events, which allows teams to personalize scrum.

Grooming the Backlog

At the beginning of the sprint, scrum teams discuss what they’ll build for the next 30 days (or less). Before the “sprint planning” meeting, though, someone has to take the potential pieces of work to be done, break them into reasonable chunks and prioritize them in order for the team to decide what they will focus on during the sprint.

Since 2003, the official Scrum Guide has referred to “backlog refinement” instead of grooming. (The old name remains popular.) Whatever the name, it’s a process by which the business collaborates with the technical staff to figure out what to build next. The process can happen continually – after the daily scrum, during the sprint, during sprint review or even during the sprint itself. It can occur continuously and informally, it can be a formal meeting, or part of someone’s role. Scrum doesn’t describe exactly how this should be done.

It’s common practice to “groom” the backlog into stories. That makes sprint planning, which we discuss next, run much more smoothly.

Sprint Planning

According to the 2014 Scrum Guide, the sprint planning meeting for a one-month sprint can take an entire business day. The event answers two questions:

• What can be delivered in the increment resulting from the upcoming sprint?
• How will the work needed to deliver the increment be achieved?

Teams use past performance, the backlog, and other data points (such as vacation plans) to figure out what they can expect to do. This is called the sprint goal. The second piece is how the team will work together. That might mean breaking the work into stories, identifying risks, creating spikes (or experiments to determine how things will work), doing design work and so on. This explanation is designed to give the customer, or economic buyer, enough confidence to let the team do its work. It’s also essential to doing the work so time invested isn’t time wasted. After all, the technical staff has to figure out how to do the work before they do it.

The Daily Standup

The single most misunderstood part of scrum may be the daily standup. Without trust, the standup devolves into a status meeting where team members prove how busy they are. The concept of a status meeting is redundant with the modern tools we have, though – the scrum board and Web-based tools can tell management about status changes on a day-to-day basis.

The purpose of the daily scrum, or standup, is to help move the ball downfield. People need to be able to ask for help, to bring problems and to offer help and advice without ego and fear getting in the way. One way to shift the focus is to talk about the work instead of the individuals. I cover this at length in the article How to Save the Daily Standup Meeting.

The Scrum Guide itself implies this in passing, claiming that the purpose of the meeting is to discuss the work that has happened since the last meeting. That said, the guide does recommend the famous “three questions” – what I have done, what I will do and how I am blocked. The purpose being to discuss how the work is flowing.

The Retrospective

The theme of scrum is inspect and adapt. This forces the team to take a break every now and again to stop and reflect. The sprint retrospective is one such break, happening after every sprint. The technical staff steps aside and discusses what happened, what went right, what went wrong and what to do differently next time.

This retrospective is similar to an after action review, postmortem, or any other “lessons learned” type of meeting. One key difference with the retrospective: Changes can always be viewed as experiments. Instead of agreeing on a new policy to happen henceforth forever for all teams, the team simply agrees to try the new technique for a few weeks until the next sprint ends, at which time it will get a chance to try something different.

The Scrum Guide lays out these three goals for the sprint retrospective:

• Inspect how the last sprint went with regards to people, relationships, process and tools.
• Identify and order the major items that went well, as well as potential improvements.
• Create a plan for implementing improvements to the way the scrum team does its work.

Now that we’ve discussed the essential ceremonies of scrum, let’s talk about the different roles.

The Team Member

Scrum separates the people who manage the process (the scrum master), those who define the work to be done (the product owner), and anyone with responsibilities to build the product (the team member). Sometimes called technical staff or member of the technical staff (MTS), team members possess different skills or abilities. The key to staffing a scrum team is that the team should have everything it needs to do the job; this may include supporting production, creating graphics or possibly even provisioning systems. Technical Staff members typically worry about what needs to be done and who has the skills to do that regardless of specific titles and job descriptions. In many cases, teams will pair up to combine/spread knowledge so many people can accomplish a task.

This focus on team members having very specific skills can create inefficiencies because some people won’t be busy 100 percent of the time as they wait for a new build or a story to be created before they are needed. Scrum, however, focuses on effectiveness (having the skills when they’re needed) over efficiency (keeping people busy all the time). Scrum teams often see slack and inefficiency, and choose it as preferable to the alternative, which is splitting the team member to support multiple projects

The Product Owner

Scrum splits the people doing the work (the technical staff) from the person who decides what to build (the product owner). He or she also defines the backlog and works with the team to understand what’s possible, how long things will take, and how to break the work into manageable chunks. Once the work is well defined – typically as “stories” and bug fixes – the product owner sets priorities, defining what the team should work on next. During the sprint, the product owner works with the team to clarify what the stories mean and, perhaps, to negotiate scope.

Schwaber has called the technical team the car and the product owner the driver. That is, the team needs to be able to get to speed, go the distance to the sprint goal and respond to changes (turn). The driver takes responsibility for where the car goes. If the driver takes the team in circles, it’s not the team’s fault.

Some thinkers about software don’t see this separation as essential. Some teams, notably two-pizza teams at Amazon and classic research and development groups, allow the technical staff to figure out what to build. Still, separating the roles, with the product owner in the marketing or “business” role, remains the overwhelmingly popular way of thinking about scrum.

The Scrum Master

Changing to scrum means more than four new ceremonies and some titles. It means a shift in thinking. The scrum master’s role is to make sure the mind shift happens and is understood. For example, if a product owner changes the sprint goal, which changes what the team works on, the scrum master would enforce the rule that changes happen after a sprint (or the rules for canceling a sprint, which are designed to be awkward and painful on purpose).

The spirit of the scrum master is a service leadership role. He or she serves the team by facilitating the daily scrum and removing impediments. The scrum master also serves the product owner by coming up with techniques to groom and improve the backlog. And so on.

The Scrum Board

The scrum board shows the state of the work: What’s being done, the status of what’s being done, and what comes next. The board can identify bottlenecks. Teams that mark stories using different colors can learn how long each story spends in each state. It’s common for teams to conduct the daily scrum meeting around the scrum board in order to “watch the flow.” The board is the status report. In The People’s Scrum, Tobias Meyer calls this the “heart” of Scrum.

Tasks and Stories

The Scrum Guide doesn’t describe exactly how the backlog is split. You could go with use cases, stories, a requirements document or a verbal conversation. However, most teams seem to standardize on stories.

A story describes a bit of functionality from the customer’s perspective. “Research new user interface” isn’t a story, but “add middle name field” is. A common convention for stories is to describe them by what they enable the user to do and why – “As a (usertype) I want to (activity) so I can (result).” Stories often include text description and acceptance criteria, which are negotiated before coding begins. As a bonus, acceptance criteria can become a major input for testing.

Early versions of the Scrum Guide described tasks, the details of “how” work, down to 30-minute increments. Add up all the tasks to 40 hours per person, goes the thinking, and you can predict results. To borrow a line from Douglas Adams, “This made a lot of people very angry and is widely considered a bad idea.”

It turns out that knowledge work isn’t predictable on those levels. The number of work-hours available to actually do story work is often less than 40. Instead of tasks, most modern teams try to make all stories about the same size and predict sprints by the number of stories. They can also use relative sizing to compare and count story points.

Empirical vs. Defined Processes

Scrum is based on empiricism – the idea that we learn what we need as we go, yet still need to make decisions to get started. This is different than a defined process where you “plan the work, then work the plan.”

By way of example, consider trying to learn to barbecue vs. baking. To bake a cake, there is a specific recipe: Add ingredients, heat the oven to a certain temperature and keep the cupcakes in for half an hour. Directions in barbecue are more likely to say, “Flip when things look right.” You need to inspect the meat and adapt your plan based on how quickly it’s browning.

Scrum accomplishes this with the idea of sprints. Build what you think needs to be built – for a while. Then show it to the customer to get ideas for what to build next.

Wicked Problems

Some software projects are that simple. Plan the work, then work the plan. For those, the waterfall might be appropriate. The problem, though, is that, sometimes, the plan is wrong, and the problem can’t be fully understood until the team has made an attempt to solve it.

This is sometimes called a wicked problem. Fred Brooks mentions this in his landmark book The Mythical Man-Month, observing, “Plan to throw one away – you will anyway.” Brooks’ idea was to move to fast prototypes. Remember, Wicked Problems, Righteous Solutions describes Scrum as an “all-at once” model, creating product increments.

More than the product model, scrum assumes the team doesn’t yet know the best way to develop. Try to develop the product in a small increment, inspect how that went and adapt that in the retrospective.

The New New Product Development Game

As noted, “The new new product development game” studies how Japanese companies built electronics products. It concludes that, instead of phases and handoffs, which draw from an (old) manufacturing analogy, the best new product development used a multidisciplinary team, with activities working concurrently. The team members that authors Hirotaka Takeuch and Ikujiro Nonaka study are autonomous: They figure out what needs to be done and do it. Development “phases” overlap, instead of occurring one at a time. This paper inspired the software development method that Sutherland and Schwaber came up with and called scrum.

Scrum Certifications

Founded in the early 2000s, the Scrum Alliance is a nonprofit essentially serving as the voice of the scrum community. That organization created a variety of credentials, including the Certified Scrum Master, the Certified Scrum Product Owner, the Scrum Professional, the Scrum Coach and the Trainer. Product owner and scrum master certifications can be earned in two- or three-day classes (along with a test); the others combine study, professional development, hours of experience and a real-time, generally face-to-face, evaluation component. The certificates get progressively harder to earn. Training for certifications is available at the Scrum Alliance’s notable Global Scrum Gatherings, which also offer Scrum Coaching retreats organized by the alliance itself.

In 2009, Schwaber left the Scrum Alliance to start Scrum.org, which uses a similar certification scheme and also hosts numerous scrum events.

How Scrum Has Changed

Scrum has changed over time. The initial goal for a sprint was 30 days, but as teams saw value in deploying to production more often and in shorter experiments, the ideal sprint length has changed. Today’s industry practice for sprint length is typically around two weeks. Likewise, tasks have gone away from the Scrum Guide, while backlog grooming has changed to backlog refinement.

Visualizing Scrum

In addition to the Scrum Guide, Ken Rubin’s Essential Scrum is another popular modern guide. He created many images for scrum in that book; we use some of those images here with his permission.