Digging into tech debt

In my discussions over the years with CIOs, they have repeatedly shared that tech debt prevents them from transforming their companies once there is a corporate will to do so. In fact, I would argue that tech debt is the big explainer for why only 28% of legacy businesses have successfully digitized. Given this, I put together a special edition of the #CIOChat to discuss this topic in detail.

What is the scope of tech debt?

CIOs include many more things in the tech debt category than I thought they would. CIOs say that it is the sum of all the maintenance you should have done but didn’t get done over the years. It extends across hardware, software, integration, networks, and security. One CIO suggested tech debt is related to builds, deploys, moves, transforms, tests, processes, and stores of 0s and 1s. All of these are candidates for tech debt pile.

CIOs suggest that the scope of tech debt encompasses specifically:

• Design debt
• Architecture debt
• Coding debt
• Testing debt
• Obsolescence debt (COBOL vs. Java)
• Deployment debt
• Operations debt
• Security debt

To put tech debt into terms that other members of the business leadership team understand, CIOs suggest comparing tech debt to deferred maintenance of physical plant and equipment. Sooner or later, CIOs suggest it will cost you. For this reason, CIOs believe the best way to measure its’ extent is as follows:

• How long does an average hardware request take to fill?
• What percentage of the team is operationally focused versus development or customer focused?
• How much of IT spend goes to ‘keeping the lights on’ versus ‘net new opportunities’?

CIOs say tech debt impact doesn’t always correlate with age. An older well-implemented solution may have less tech debt than something deployed yesterday that is sub-optimal. Infrastructure and for that matter, data and application integration accounts for a disproportionate share of tech debt, because the business understands and funds applications more readily, while infrastructure and integration effectively ‘rusts’. CIOs, for this reason, suggest that the longer a system stays around the more likely that it has a custom integration that was “judicious” at the time it was done, but years later is extra baggage that’s become critical today.

CIOs share openly that tech debt can be a drag upon time-to-market when you prioritize debt eradication as part of an initiative. Some CIOs suggested here an interest rate view of tech debt. These CIOs feel that it is a powerful and understandable communications tool for explaining what happens when you don’t pay tech debt down. It is important for CIOs to talk to their boards and CEOs about the importance of governance and at the same time, to be candid about the material impacts of weak processes. These are the things that can be fixed going forward separately from the pile of aging code and rusty infrastructure predecessor CIOs created. CIOs are clear that incurring tech debt should only be okay as a temporary solution to get to market faster. But CIOs need to have a plan/budget to remediate it as time permits.

How does organizational or IT culture exacerbate tech debt?

Tech debt represents the enterprise’s fat and cholesterol. CIOs say the more engaged the enterprise, the less tech debt there will be. CIOs say as well that while a culture of continuous change and exercise is healthy, there is no magic pill for curing all tech debt.

CIOs believe that organizational culture can contribute to the problem through misaligned funding. An example is keeping old software because it is familiar when, to be competitive something else is needed. CIOs, however, believe in the age of disruption the loans are in the rears and the debt collectors are at the door. Culture is critical to doing something about the growing pile of tech debt. Is your organization interested in just adding to what it already has or in doing the right thing? CIOs claim the biggest IT culture hurt is the quick elimination of innovative ideas that really eliminate debt, usually done at the lowest organizational level.

If the culture of the organization is ‘maintenance and punch list’, you must do a major resourcing model immediately. Otherwise getting going will be extremely hard. If IT is always behind and always ‘in trouble’ with the business for missing deadlines, the culture will push to get the next deliverable at all costs. In this case, CIOs say a CIO/CFO/CEO/Board discussion is needed.

CIOs clearly need to influence allies especially the CFO to pragmatism on the total cost of ownership for technology. If you have a culture of only rapid prototyping that mysteriously becomes production, you will accrue tech debt at a high velocity. The impact for this must be defined in business value terms. Examples include drag on time-to-market or pain to customers. It is critical for CIOs and IT teams be able to articulate the value of tech debt reduction investment in business terms.

How important is enterprise architecture to limiting tech debt?

One CIO said you wouldn’t build a house without a plan or design? For this reason, they suggest it is critical to bake in the agile governance. Part of this should ensure that 30% of the prioritized backlog over a release/quarter/year needs to be applied to addressing tech debt.

You should show the past and how you will change the curve as part of a new enterprise architecture. Otherwise, your organization will not learn. An effective enterprise architecture team should as a goal accrue as little tech debt as possible and enable paying off existing tech debt quickly. CIOs like to compare the traditional, tech debt-ladened organization to the “The Winchester Mystery House“.

CIOs suggest it is the enterprise architecture that should have the mechanisms in place to continuously monitor the benefits of existing assets and flag the ones that have high potential. Clearly, many smaller organizations, don’t have enterprise architecture teams. But they should, nevertheless, have an enterprise architecture. They need to define reference architectures and prioritize what tech debt to focus on fixing.

The fact that tech debt is typically distributed across the technology portfolio. This means resolving it requires a team or distributed effort not just something for the architecture team. One CIOs says this like asking the plumber to make certain the electrical wiring is up to code. The fact is most of it is badly managed. But CIOs need to make sure that it is overseen. Where tech debt is a short-term fix, everyone needs to understand it’s like borrowing money. In other words, there capability and a plan to pay it back.

CIOs suggest that it is critical for enterprise architects to limit future tech debt by designing systems that can be properly maintained and operated with existing staff and tools. CIOs suggest that sometimes fixing things can start by creating a visual for the enterprise architecture. For example, the business has each of the upstairs rooms. The back office has the main floor and technology has in the basement. Network is all the plumbing.

Many IT departments fail to develop a body of knowledge on what they’ve learned in enterprise architecture, app development, integration, operations, ITSM, and customer service. They make much the same mistakes year after year. Agile retrospectives can be used to address this. It is important, for this reason, that CIOs work hand in hand with the enterprise architects. Everything needs to be articulated in business value terms. CIOs need to empower teams to prioritize tech debt eradication in project backlogs. Although enterprise architecture may seem like table stakes that the management team won’t fund, CIO better figure how to do this.

Where should CIOs and EAs start to reduce the pile of tech debt?

Decisions about tech debt need to be made with business growth in mind. The primary reasons for fixing it, in many cases, can include the continuing existence of the business. CIOs should start with the biggest hinderance to the business. To address tech debt in a CI/CD fashion, you need to do the following:

• Make it everyone’s problem (put it on the balance sheet)
• Catch up (request boards invest for several years)
• Measure + control ongoing tech debt with data-based approaches

CIOs should start by understanding each piece of tech debt and overlay fixing them on the future of state of the company based upon business strategy. They should, also, determine which still have value along with all the integration points to other things including outside service partners. CIOs and EAs should start reducing the size of the pile of tech debt with business functions that are no longer relevant.

The biggest business value opportunity in reducing tech debt is likely in the human part of the problem. Meanwhile, the extent that microservices are implemented, self-healing APIs can reduce the need to monitor for tech debt. CIOs need to prioritize by risk of breach or brittle system going down. The latter includes what the business wants that the current application stack can’t deliver. CIOs suggest that it is important to avoid too much strategic plan evaluation or even worse prognostication. A CEO/CIO partnership is clearly critical to overcoming these potentials.

CIOs believe that IT departments need to get started. Pick something identifiable, fundable, and with measurable impact. You should then deal with it, make a big deal out of getting it done. Whatever you do, don’t do months’ worth of analysis just to confirm the debt you already know you have. One CIO said here that they would guess that the issue looks so strange and so large on paper. The problem is that it is big.

CIOs say fixing things starts by IT leaders finding a way to explain the breadth and depth of the tech debt. This is a problem that the board needs to get and support fixing. This is another reason for having Qualified Technology Expert Directors on the board

Some CIOs say that they prefer to do a quick analysis. They can then start looking for quick wins while they drill down on the people, culture, technology strategy, and roadmap. CIOs increasingly want to make sure they are building a ‘LEGO based architecture’ to protect the future and deploy wherever possible low code or no code. One CTO recently suggested even that low code or no code teaches good habits to initial developers.

Should application modernization have tech debt reduction goals?

CIOs insist that just about everything IT organizations do should have process efficiency and effectiveness embedded into them. CIOs believe, for example, that application modernization efforts should include fixing related tech debt.

What architectural and ownership goals can we establish that allow us to more effectively leverage technology in support the business. Clearly, one of the big reasons to move to hyperscale cloud has been its impact on Infrastructure tech debt.

CIOs believe that all technology plans and budgets should explicitly consider tech debt. It seems clear that tech debt doesn’t typically get better with age. All of this relates to business value, current state, and complexity and priority. CIOs suggest that before an IT organization builds, buys, or rents software or data, they should ask several questions:

• Are there local/3rd party APIs?
• Is there a master data for it?
• Is there a schema you can extend?
• Are there existing SaaS options that you can build on?

CIOs are clear that tech debt eradication should be in the budget for each application modernization effort. The smaller the debt, the more modernization dollars that should be provided. In justifying this position, it is important to make sure that everyone understands and is aligned on “debt metrics”, otherwise there will be a lot of finger-pointing and frustration.

CIOs believe that tech debt in general is a bad thing, but they also believe it is hard to avoid completely. It seems clear that enterprise architecture can help. More importantly, CIOs suggest it is time to wake up the organization. And while CIOs believe that the business and accounting sides may want nothing to do with eliminating it, they also believe ignoring the growing pile leaves their organizations at peril as world digitizes to the disadvantage of legacy businesses.