Every software project begins with big dreams and grand visions. Somewhere in an alternative universe, there is a project that fulfills every dream but all too often software projects in our universe stumble toward the finish line and sometimes cross it.
Of course, by definition, software project failure isn’t a binary thing. You can end up with code that runs well but no one uses. Or you can end up with code that won’t even compile. Sometimes you can salvage something useful from the flaming wreckage and sometimes it’s best to run away before it explodes.
When the smoldering mess cools, the autopsies begin, as people want to know what went wrong. Here are the most common culprits.
Too few team members
Trying to do too much with too few programmers is a common problem. Developers can only grind out so much code before they burn out. I once worked on a team where the manager thought the right way to squeeze more work from agile teams is to schedule each “sprint” so it began immediately after the previous one. No thoughtful pauses to figure out what was working and what wasn’t. Sprint 42 ended Wednesday at 1:59pm and Sprint 43 began on Wednesday at 2:00pm. Retrospective analysis meetings were scheduled after the next sprint already started. Some clever guy suggested they be renamed “marathons” and soon found another job.
Of course, it’s hard to know how many programmers is enough. Sometimes roadblocks and problems get in the way. It may not be the manager’s fault that the job doubles in size but if you don’t have enough people on the job, your project is likely doomed.
Too many team members
If too few programmers can be bad, too many could potentially be worse, as network effects can doom a software project. More people means more coordination and that means more meetings, taking time away from writing code. But if you don’t hold enough meetings, you’ll soon discover Team A’s API doesn’t interface Team B’s microservices.
It would be nice if we could just throw money at a problem by overstaffing a project, but you can’t.
Too much communication
Writing software is a solitary art. Humans can work together but only in limited bouts. Many developers hate meetings because they need to shift their mental gears from the deep immersive logical thought into the more open, social mode. This takes time. Some team leaders try to fight failure by holding more meetings to keep everyone synchronized. It’s a noble effort but you can hear the gears grinding. The team needs to share enough information to stay synchronized but any more just wastes brain cycles.
Fundamental feature changes
In theory, developers like to consider themselves agile. That’s why they embrace the word. But sometimes agility can throw everyone off balance. It all depends whether the shift requires fundamental changes to the underlying framework. It’s easy to be agile when moving a button around or changing a color. But when it involves reworking the database schema or mucking around with sharding and replication, there’s no easy way to gracefully pivot.
Picking the wrong technology for the job
Even if you plan carefully and draw up the correct list of features, things may fail if you use the wrong technology. Databases, for instance, are designed to be general and flexible, but do have architectural limitations. Push them to deliver something they’re not designed to do, and they will slow to a virtual halt when asked to scale. Or you may start off with a NoSQL database because they sound cool only to discover later that you really need ACID-grade transactions to keep things consistent and the database doesn’t offer them. Oops.
Good planners draw up a list of features and prioritize them. But sometimes priorities don’t line up with the reality of implementing them. In the worst cases, the most important features are the hardest to create.
What are your developers to do? If they concentrate on the most important feature, they will make no progress and may end up delivering none of the functionality. But if they start knocking off the easy ones, they may end up with something that’s worthless.
Good planning requires more than a checklist. Architectural vision must take into account the needs and cost of delivering them.
The market window closes
Sometimes it’s not the programmer’s fault. One of my projects was to turn a best-selling reference book into an app. The book sold like hotcakes in the years before the internet. The plan was to tap into that demand and make an interactive version that would let people sort and search through the data. The programming team delivered software that included everything in the book but was faster, prettier and much lighter than the book itself. But no one wanted it anymore. There were enough other sources and no one needed another app that did much the same thing as news sites do everywhere.
Sometimes an idea seems great, but the marketplace has moved on.
Bad architectural decisions
On one project, I was given the job of changing one number in one row in the database. When the user finished registering, I was to add the user’s id number to the latest order. Sounds simple, right? But the system was built on a microservices architecture and I couldn’t solve this by writing one line of code that would tell the database to UPDATE that column. Nope. The architectural plan was to add a new microservice call to the existing stack and even this was hard because my first microservice call needed to trigger another microservice call and so on.
In the end, the architectural whiz who created this network of microservices told me that it was all very simple and outlined a serpentine path through five layers of the architecture. My job was to add five new API calls to five different microservices, which also meant adding five sets of automated tests for each layer. And each API was developed by a different team over the years, requiring me to understand and emulate five different styles of coding. All to change one number.
Architectural decisions can last a lifetime — especially if your ego is thoroughly invested in them and you can’t change them. Project managers have to be ready to notice when the main architectural plan is not working so big decisions can be made.
Blaming political factors for a technical failure may seem weaselly, but it’s increasingly true. As projects grow bigger and span multiple organizations, it shouldn’t be a surprise that factions appear and groups jockey for control, resources and ultimately power.
Political factions are different from real technical differences. There are often technical standards or code bases that do much the same thing in different ways. Take XML and JSON. Now that I’ve typed that, I can feel the fans of either rushing to explain why they aren’t the same and their favorite choice is the right one. But when one part of a team loves one choice and another holds the competing faction in the highest regard, well, friction is going to tear them apart.
This has grown even more common as architects split up applications into multiple, smaller services and APIs. Different groups will end up controlling these and they won’t always get along. If Group A likes JSON and Group B clings to XML, well, your team will either need to implement both or get one of them to change. All three are a pain for any team that must work with both Group A and Group B.
Betting on technology that’s not ready for production
Programmers love the latest tools and frameworks. They want to believe the new approach will sweep away all the cruft that bogs down the previous generation.
But often, the next generation isn’t ready for production use. The new features may seem perfect, but there are often gaps that aren’t immediately obvious. Sometimes the code supports only a few file types or interfaces with only a few databases. The others are coming soon, they assure you, but your project needs to ship this month and “soon” could mean six or more months before the features you need are complete.
Betting on technology that’s soon to be outdated
In my experience, the old stuff is usually more reliable and battle-tested, but that doesn’t mean old technology is perfect. Features can be missing that are vital to your software project once it goes live. Worse, betting on old tech can cause you to miss out on future opportunities based on changes down the line. New ideas, protocols and file formats appear, and they can go unimplemented. And if someone from a competing team insists that you support some new protocol then the old tech will hurt.
Deadlines are tricky. Many projects need to make it to market by a particular season or event. Yet when deadlines are first written down, your developers haven’t begun to discover the roadblocks and hurdles in their way. Then if the project slips and the event passes without the software being launched, the entire project is seen as a failure even if the code is just about to run smoothly. Deadlines help everyone focus and pull together, but they also create expectations that can be unrealistic.
A good product manager surveys the competition before diving in but no one can predict what competition may appear out of nowhere. If new competitors introduce new features that you must duplicate, well, see the sections on feature changes and priority mismatches, above.
Rushing the process
Many software projects start as the vision of a person or team who wants to fix something. They come up with a phrase like “Snapchat for Y” or “Uber for Y” and then expect the product team to be as responsive as Snapchat or Uber. The problem is that figuring out the scope of the project, sketching the data flows and imagining the UI are often ten times as much work as writing the code. But the imagineers want to go from idea to code right away.
The wireframes, database schema and user stories aren’t just hand waving, but an essential part of the job. But most people want to believe that a software project is just writing code to implement an idea.
False belief in the power of software
Dreamers often have unrealistic beliefs in the power of software to change the world. Many imagined that social media would unite us but somehow it’s just exposed fault lines that were always obvious. Software projects often begin with slide decks that promise to revolutionize some corner of the world. When shoving bits in a database doesn’t transform anyone, well, people get angry, bored, confused or worse. They say the software is broken because it fails to deliver the magical transformation that everyone expected.
Many software projects can compile, pass QA, ship and even get decent reviews but ultimately fail to achieve any of the promises on the slide deck because, well, those change-the-world promises are impossible.
We love the vendors that produce the libraries and tools that enable us to create magic with just a few lines of code. Occasionally, they get wind of their power and use it to destroy a project. The budget sheet for version 1.0 was so good that the management didn’t hesitate to approve version 2.0. Then the vendor decides to squeeze us by tripling or quintupling the price.
This effect can be felt even when the vendors don’t do it on purpose. Free tiers can make a project look very cheap. Then when demand soars and the second version expands the demand, the real prices kick in.
In a year with pandemics and protests, nothing has changed faster than the zeitgeist. Did the project make strong privacy protections a central feature? Whoops. The pandemic has made everyone interested in contact tracing. Did someone want to focus on business travel? Whoops. The hotel industry has collapsed. Bigger software projects that can take a year or longer risk being upended by cataclysmic events. What seemed like a wonderful idea at the beginning may seem hopelessly lost and disconnected when it’s time to go live.
It’s not just changes in the world at large. Tidal changes in the tech community can have the same effect. NoSQL was once a genius idea that would liberate us from relational schema. Then someone realized the files bloat because every record carried a local schema. A good agile development team can shift when technological sea changes shift the attitudes of the leadership and the customers. But even the most agile teams can’t deal with big shifts that suck all the air out of their architectural plans. The system was built assuming that X was a great idea and suddenly X is dirt. Sometimes it’s best to blow it up and star again.
Too many additions
Some software projects start well and even ship successfully. Then someone adds an extra feature or three, grafting new code on to the existing version in a way that the code continues to limp along. Heroic developers may pull this off several times, especially if the initial architect planned well. But at some point, the foundation crumbles. Maybe the database can’t handle the load. Maybe there are too many JOINs needed to satisfy the various queries. Good software can get too fat, sometimes thanks to little improvements that pushed it over the edge.
Moving goal posts
The initial plans called for a database to tracks customer spending to help with marketing plans. Then some genius added a feature that uses artificial intelligence to correlate spending with the weather forecast. Or someone else wanted the software to automatically bid for search engine ads. Changing the destination can upend a project.
It’s rare for the changes to ruin things on their own. But new goals can reveal weaknesses and trigger failure modes. Maybe the team is now too small to complete the project successfully. Maybe the technical foundations are wildly inefficient for the new approach. It’s hard for everyone to anticipate the fretful combinatorics when the decision is made to change the goals.