Lesson #9: To maximize team efficiency, the project plan must consider testing efficiency. This may determine feature implementation order.

The software is buggy. The test team tests around the areas that aren't implemented or aren't working, but it finds a number of issues that block further testing.

Even worse, when the test team gets a bug fix from development, 30 percent of the time it doesn't fix the problem. In this state of code churning, the project hurtles past the deadline. The PM is pressured to ship (he wants his trip to Tahiti too!). The developers and testers are told to increase their efforts, work together to achieve the goal, do whatever it takes...

Lesson #10: Buggy software takes longer to ship.

The product ships in an unknown state. Last-minute functionality was added, and it received only cursory testing. A large number of identified bugs are still open, although all known critical problems were either addressed or reclassified as "serious." The maintenance release is already being planned. The team is exhausted. It worked heroic hours, again, and produced a barely supportable product—again. The customer is unhappy—again. The product has features the customer doesn't want or doesn't understand, and it's missing several major items they were expecting. Accolades come down from above for another "on-time" delivery.

What went wrong?

• Management doesn't recognize that "on time" doesn't equal "satisfied customers."
• The entire project team is driven by schedule. Every decision shows schedule-, rather than quality-consciousness.
• Shortcuts taken to improve schedule time (including unfinished requirements, insufficient system design, no unit test) actually made the project take longer to complete.
• The maintenance release is, in reality, still the primary release, but now the unhappy customer is involved too.
• People are exhausted, burned out and not utilized effectively. The rewards system is messed up!

Six months after this project shipped (and eight maintenance releases later), an analysis is done to determine the origin of all the bugs. The analysis shows:

• 50 percent of the bugs were introduced in the requirements. These were due to unclear and vague requirements, as well as functionality that was not defined, and thus had to be introduced in a maintenance release. This also includes data issues and equipment issues where the test team didn't have the right data or equipment to reflect the customer's environment. Additionally, all bugs associated with the unwanted features are counted here, since those bugs would not have occurred if the features hadn't been implemented.
• 15 percent of the bugs were due to design issues, particularly interfaces between code modules and the database.
• 25 percent of the problems were coding errors, both in new code and regressions introduced in the fixes.
• 10 percent of the problems were system integration issues that were visible only in the fully integrated environment.

Continue Reading