10 reasons software projects are slower than we expect

Douglas Hofstadter, longtime computer programmer and author of Gödel, Escher, Bach, summed up his experience in the computing trenches with this eponymous law: It always takes longer than you expect, even when you take into account Hofstadter’s Law. 

Yet organizations, stakeholders and project managers the world over still put timelines on software projects. Even programmers themselves. Is Hofstadter’s Law more self-evident to programmers, who must grapple firsthand with the fussy literalness of machines, or to project managers, who must try to lead everyone through the slog? Project managers count on the API to save two months of work, but it’s the developer who discovers that the API has 129 options — except the one the company needs. Getting your code to build is difficult, but so is managing eight programmers who are all struggling with different bugs that stop the code from building.

From the outside, project sponsors drum their fingers on conference room tables, impatiently. Meanwhile, the suits and project managers see programmers frantically paddling and getting nowhere, and the coders see only taskmasters who can’t understand how strong the current is.

Long ago, someone tried to be scientific about measuring productivity and counting lines of code. Fredrick Brooks tore that theory apart in 1975 with The Mythical Man-Month, a book that made it clear that all those mathematical models and assumptions of linear progress just don’t apply. 

Ever since, the thorn bushes along the path to project completion have only gotten thicker. Clever ideas about agile this and continuous build that may work temporarily on a local problem, but they make everything more complex and complexity is the ultimate problem. Customers, stakeholders and stock holders all demand more and more, but more is just going to take longer. 

Here are 10 reasons why software projects take longer than expected, despite new tools and techniques meant to accelerate software delivery. They may not explain every slow down, but they can help managers understand just where all the coding effort is going.

Scope creep

You might use the same name for the project the software team launched months ago, but the scope and deliverables have changed so much that it might as well be called something different. The overarching goal may remain the same, but someone changed the architecture enough that everything needs to be redone from scratch.

Perhaps it’s not fair to think of it as a project that’s taking longer than expected — more that the first project “ended” a while back and a completely new one has been launched, under the same name and without the usual project launch fanfare. In fact, there may have been a few other projects “completed” in the intermediate months, and thus an accurate accounting would not indicate that your ONE project has taken many more months than originally projected, but that your developers have spent those many more months on many projects all with the same name.

Deliberations

Despite software developers’ grumblings about open offices and the need for isolation tanks and gigantic headphones to get down to coding, somehow there’s not enough time in the day for all of the talking that needs to go into software design. It’s almost as if we find debating the approach is more satisfying than completing the work.

Of course, we can laugh about the amount of time we spend in meetings chattering about the best way to structure some menu or design some schema, but the truth is, even the simplest project requires building something fairly new. Coding isn’t like building a road, a task that is older than the Roman Empire. It’s going to take longer to optimize software because it hasn’t been done many times before — and certainly not to these specs on these systems for these business needs.

Agile has limitations

For manager types, process is everything. So when it comes to keeping pace, frameworks and methodologies are the manager’s first best solution. Software project managers like to argue about the best development methodology but they all have limitations. Frameworks, after all, are merely guidelines, and work in action is often different than work in theory, resulting in exceptions that must be navigated, even as the project timeline ticks on.

As for developers, we tend to gravitate toward agile because it gives us the most freedom to pick and choose where to contribute, but it can lead to chaos without a guiding hand. One cyclical manager told me not to worry whether the code would be right because we could always create a new agile ticket to fix it. Voilà! We get twice as much credit for solving twice as many tickets. It’s almost as if we accomplished twice as much.

No one can foresee enough

Isaac Newton, the so-called inventor of calculus, once said, “If I saw further than other men, it was because I stood on the shoulders of giants.” Calculus may be the pinnacle of high school mathematics and the foundation for many sciences, but it only has a few basic operations. It’s not like a payroll system that must work with 50 different states not to mention the District of Columbia and Puerto Rico.

The point is that we software developers are called on to build systems that unify dozens if not hundreds of communities and it’s almost impossible to keep all of their needs in our heads. To make matters worse, there are often no giants with shoulders to stand on because the new system is going to, like, totally disrupt everything that came before while being super cool at the same time. So we gamble on some basic abstractions, try to fix details as we iterate, and then hope for the best. Fixing all of the objections will take time because there are so many details.

Stakeholders change their minds

You may think software developers control the pace of their work, but the truth is, we’re often working with moving targets. It’s not fair to blame the software team for many of the changes created by the customers and other stakeholders in the organization. They’re constantly bringing new requests for modifications and enhancements. Sometimes adding one little thing can be very easy but sometimes it can be very hard and ruin much of what was accomplished before.

Data structures are rigid

Data is a tricky beast to tame and much of the work on software development goes into herding the bits into the right stables for the night. One of the great tensions is building the data models and ensuring that the databases store the information accurately and completely. On one hand, we want the data models to be strict so that, say, we can trust there will only be nine-digit numerical Zip codes in the Zip code field. On the other hand, we want the models to be flexible so we can just wave a magic wand and let someone register from Canada, the place where the postal codes are mixtures of letters and numbers, but in a very strict pattern. Iterating on these models will take time.

We need the structure for them to work — but changes can invalidate the structure. It’s an eternal battle. The power is that there’s one entry per person. But once you admit that you want to track a person after a name change, there are two entries. And then it becomes both more and less at the same time.

Falsehoods arise

Some programmers like to catalog simple statements that seem true until you try to render them into code. These falsehoods include statements like, “There are 24 hours in a day.” That seems obvious, except for the days when daylight savings time begins or expires. Those days change each year and are completely different in Europe. In other words, a day could have 24 hours on one continent but 25 on a different one.

You will almost certainly discover several new falsehoods to add to the list with each project. We can only hope that the new ones will be easy to work around and the users won’t be too upset when they stumble across one of the inconsistencies.

Complexity kills

Programmers want to streamline the systems they work on but users and stakeholders want to “improve” the systems by adding additional cases and functionality in pursuit of possible business value. Features are great, but adding smarter and better features means adding logic and sometimes the complexity of the logic explodes exponentially.

Some features are easy to add but some have unintended consequences that trigger strange surprises that lead to more unanticipated meetings to make decisions you never imagined you would care about. If you’re not careful about adding another layer to your code or another extra column to the database, you may end up surprised when the complexity of the project explodes and you’re further from the finish line than you were just yesterday. 

Design review invites feature creep

Design review is an essential part of the development process, only everyone wants their software to do more and thus love to add feature requests to meetings designed to review the design. Sometimes it’s possible to deny the requests that are obviously beyond the scope of the current project, but unfortunately there are plenty of seemingly innocuous requests that require large amounts of recoding. Sometimes it can take several hours of code review just to figure out whether it’s one of the easy cases or one of the hard ones.

But we can’t get out of reviewing the design with stakeholders. It’s part of a good development process. So we’re doomed to rattle around the code trying to decide just what we can add and what we’ll leave to version 2.0 or maybe version 5.0.

Slower is better

While you may get steamed at slow project progress, the pokey nature of software development is not a bug — it’s a feature. Rushing through important work is just a way to guarantee that it will turn out flawed and faulty. When coders take their time, we ensure that the foundation will be solid and the logic will be well-reviewed and thorough. Complaining about the pace just distracts us from our appointed rounds, adding risk to the project. Just let us get back to work.