Empowering Product Development
By Pete Deemer
Sep 10, 2007

The traditional way to build software, used by companies large and small, is commonly known as The Waterfall. There are many variants, but it typically begins with a detailed planning phase, where the end product is carefully thought through, designed, and documented in great detail. The tasks necessary to execute the design are determined, and the work is planned using tools like Gantt charts and programs like Microsoft Project. The team arrives at an estimate of how long the project will take by adding up detailed estimates of the individual steps involved. Once stakeholders have thoroughly reviewed the plan and provided their approvals, the team starts to build. Team members complete their specialized portion of the work, and then hand it off to others in production-line fashion. Once the work is complete, it's delivered to a quality assurance organization, which completes testing prior to the product reaching the customer. Throughout the process, strict controls are placed on deviations from the plan to ensure that what is produced is actually what was designed.

This approach has strengths and weaknesses. Its strength is that it's supremely logical: think before you build, write it all down, follow a plan, and keep everything as organized as possible. It has just one great weakness: humans are involved and humans don't work very well this way.

For instance, this approach requires that all the good ideas come at the beginning of the development cycle, where they can be incorporated into the plan. But as we all know, good ideas appear spontaneously throughout the process - in the beginning, the middle, and sometimes even the day before launch. With the Waterfall approach, a great idea late in the development cycle isn't a gift, it's a threat.

The Waterfall approach also places a great emphasis on writing things down as a primary method for communicating critical information. The very reasonable assumption is that if I can write down on paper as much as possible of what's in my head, it'll more reliably make it into the head of everyone else on the team. Moreover, if it's on paper there's tangible proof that I've done my job. The reality, though, is that most of the time these detailed 50-page requirements documents just don't get read. And that's probably just as well, because when they do get read the misunderstandings are often compounded.

Something else that happens when you've humans involved is the 'hands-on aha' moment - the first time that you actually use the working product, and you immediately think of 20 ways you could have made it better. Unfortunately, these very valuable insights often come at the end of the development cycle, when changes are most difficult and disruptive - in other words when doing the right thing is most expensive.

Humans also have a poor ability to predict the future. For example, the competition makes an announcement that wasn't expected. Unanticipated technical problems crop up that force a change in direction. Furthermore, people tend to be particularly bad at planning things far into the future - guessing today how you'll be spending your week 8 months from now is something of a fallacy, and it's been the downfall of many a Gantt chart.

In addition, the Waterfall also tends to foster an adversarial relationship between the team-members who are handing work off from one to the next. And this gets us to another observation about the Waterfall - it's not that much fun to work within. In fact, we'd go a step further and say that the Waterfall is a cause of great misery for the people who build products, and the resulting products fall well short of expressing the creativity, skill, and passion of their creators. People aren't robots, and a process that expects them to act like robots often results in unhappy people.

A rigid and change-resistant process will also tend to produce mediocre products. Users may get what they first ask for, but is it what they really want once they see the product begin to emerge? By gathering all the requirements up front and having them set in stone with little chance of change, the product is condemned to being only as good as the initial idea, instead of being the best it could be once the team knows more about the possibilities.

Many users of the Waterfall experience these shortcomings again and again, but it seems that in such a logical approach the natural reaction is to turn the blame inward: if only we did it better, it would work; if we just planned more, documented more, and resisted change more, everything would work smoothly. Unfortunately, many teams find just the opposite: the harder they try, the worse it gets.

The Agile family of development methodologies was born out of a belief that an approach more grounded in human reality would yield better results. Agile emphasizes building working software that people can get hands on with quickly versus spending a lot of time writing specifications up front. It focuses on small, cross-functional teams empowered to make decisions versus big hierarchies and compartmentalization by function. The focus is also on rapid iteration, with as much user input along the way as possible. Often when folks learn about Agile, there's a glimmer of recognition - it sounds a lot like back in the start-up days, when we 'just did it'.

One of the fastest-growing Agile methods is Scrum. It was formalized over a decade ago by Ken Schwaber and Jeff Sutherland, and it's now being used by companies large and small - including Yahoo!, Microsoft, Google, Lockheed Martin, Motorola, SAP, Cisco, GE Medical, CapitalOne, and the US Federal Reserve. Many teams using Scrum report significant improvements, and in some cases complete transformations, in both productivity and morale. Scrum is simple, powerful, and rooted in common sense.

Scrum organizes product development in cycles of work called Sprints, which are typically 1-4 weeks in length, and which take place one after the other without interruption. The Sprints are of fixed duration - they end on a specific date whether the work has been completed or not, and are never extended. At the beginning of each Sprint, a cross-functional team selects items from a prioritized list of requirements and commits to complete them by the end of the Sprint. During the Sprint, the deliverable doesn't change. Each work day, the team gathers briefly to report to each other on progress, and update simple visuals that orient them to the work remaining. At the end of the Sprint, the team demonstrates what they've built and gets feedback which can then be incorporated in the next Sprint. Scrum mandates that the team produce product at the end of the Sprint which is really done. In the case of software, this means code that's fully tested and potentially shippable. Scrum formalizes a new role on the team - called the ScrumMaster - who facilitates the team's use of Scrum, and serves and protects the team from outside obstacles and interference.

Scrum isn't a process. Rather, it's a framework which enables the team and stakeholders to inspect and adapt, offering flexibility and adaptability balanced by the stability required to product demonstrable results. Scrum works by making visible the dysfunction and impediments that are impacting the team's effectiveness, so that the team and organization can address them. For example, most teams aren't good at estimating how much they can get done in a certain period, and so will fail to deliver what they committed to in the first Sprint. To the team, this feels like failure. But in reality, this experience is the necessary first step toward becoming more realistic and thoughtful about their commitments, and also being even more committed to delivering what they signed up for. This pattern - of Scrum helping make visible dysfunction, enabling the team to do something about it - is the basic mechanism that produces the most significant benefits, which teams using Scrum experience.

One very common mistake teams make, when presented with a Scrum practice that challenges them, is to change the practice, not change themselves. The good news is that while Scrum is often very challenging to the team, the benefits tend to be visible by the end of the first Sprint, leading many new Scrum teams to exclaim: "Scrum is hard, but it sure is a whole lot better than what we were doing before."

The benefits of Scrum reported by teams come in various aspects of their experience. At Yahoo!, we've migrated nearly 75 projects to Scrum in the last 24 months, totaling almost 700 people. Once each quarter, we survey everyone at Yahoo! using Scrum (including product owners, team members, ScrumMasters, and the functional managers of those individuals) and ask them to compare Scrum to the approach they were using previously. Some summary data is presented here:
* Productivity: 68% of respondents reported Scrum is better or much better (4 or 5 on a 5-point scale); 5% reported Scrum is worse or much worse (1 or 2 on a 5-point scale); and 27% reported Scrum is about the same (3 on a 5-point scale).
* Team morale: 52% of respondents reported Scrum is better or much better; 9% reported Scrum is worse or much worse; and 39% reported Scrum is about the same.
* Adaptability: 63% of respondents reported Scrum is better or much better; 4% reported Scrum is worse or much worse; and 33% reported Scrum is about the same.
* Accountability: 62% of respondents reported Scrum is better or much better; 6% reported Scrum is worse or much worse; and 32% reported Scrum is about the same.
* Collaboration and cooperation: 81% of respondents reported Scrum is better or much better; 1% reported Scrum is worse or much worse; and 18% reported Scrum is about the same.
* Team productivity increased an average of 36% based on the estimates of the product managers.

Building great products and services in the highly competitive Internet market is extremely challenging. Without Scrum, Yahoo! and a growing list of other companies wouldn't have neared the degree of adaptability and continuous visibility that success in this environment demands.

By Pete Deemer, chief product officer (India Research and Development) of Yahoo! and Gabrielle Benefield, senior director (Agile Development) of Yahoo!