Technical debt and its impact on DevOps

Many companies think that DevOps practices will help them get rid of their technical debt. In general, this might work, but one must understand the situation in full.

DevOps is not a cure for technical debt. It is only a method of minimizing the accrual of new debt and freeing up the time and resources that must be devoted to removing the existing issues. However, this requires a responsible approach and an in-depth understanding of the existing technical debt, the reasons behind its existence and the right approaches to its removal.

Technical debt — a trade-off of quality for speed

What does the “technical debt” term actually mean?

“Technical debt is a shortcut in project implementation, where the code quality is sacrificed to speed, so the features are delivered on time but use the subpar code”

This does not imply that technical debt is generated by inexperienced or lazy programmers, who write bad application code. In the majority of cases, the situation is quite opposite — technical debt is generated by good programmers, who know how to write clean code but write the “dirty” code to meet the deadlines. Almost all legacy code can be classified as technical debt, which is the reason for innumerable jokes.

Usually, technical debt is reduced by later refactoring, but it is quite often left as is if it does not impair the functioning of other product features. The problem is that even minor cases of such dirty code later become the roots of huge chunks of subpar code, and refactoring them requires a significant amount of effort. In addition, if such code is hidden sufficiently well enough, it might remain undiscovered for very long and pass the manual testing pipelines without notice.

Why DevOps CI/CD pipelines don’t help remove technical debt

When the company decides to undergo the transition to DevOps, the management is often convinced that implementing CI/CD pipelines (especially for automated testing) helps discover and remove the technical debt. This is true to a degree, and this is why the usual approach to the legacy code during the digital transformation is not lifting-and-shifting it to the cloud, but replacing it with cloud-native alternatives. This requires significant investments and is a daunting task for many businesses.

However, quite a lot of legacy code might work in so-called “blobs” that pass the CI/CD pipeline unopened and unchanged, so they do not affect the results of testing while impairing the product performance in production. Such self-encapsulated pieces of technical debt are like landmines, waiting for a careless developer to stumble upon and wreak havoc, causing the product/service downtime.

The reason for this situation lies in the simple fact that no software delivery methodology or approach like CI/CD can rectify the technical debt, as its reasons extend beyond the scope of any methodology.

What are these reasons? If the developer knows that a bit of dirty work saves him lots of time — he will do it. If the testing discovers the technical debt, but it does not impact the testing results — the QA will pass it. If the company knows about this technical debt, but the cost of its refactoring is greater than the effort invested in creating it — this technical debt will remain in place regardless of the software delivery approach used, be it Waterfall, TDD, Agile or LEAN.

To elaborate further, even DevOps is not impervious to technical debt. DevOps is a set of practices aimed at writing an immutable infrastructure as code to enable continuous integration and continuous delivery of new product features while keeping the product always available. When the processes of testing and deployment are governed by code and deadlines are always short, the temptation to make a quick and dirty fix is enormous. First, the DevOps engineers say they’ll “fix it later”. Later, they don’t even try to fix what works.

The true cost of removing the technical debt: the transformation of the company culture

If technical debt becomes included in newly-implemented CI/CD pipelines, it can become almost as difficult to address as the legacy code the DevOps practices had to replace. In addition, most DevOps tools are open-source and might have their own technical debt, while many DevOps teams are lax and don’t keep their toolkits up-to-date.

Don’t forget that a typical DevOps transformation does not begin from scratch. Usually, there are lots of codebases, tools, and infrastructure that need to undergo the digital transformation and become operated through DevOps workflows. This means lots of existing dirty code can be transitioned to DevOps pipelines by its authors, who fear punishment for their laziness.

If the originators of bad code had already left the company — by the current software engineers, who might earnestly think that the things must be done this way, and this way only (or don’t wish to invest much effort into correcting the bad code). Thus said, sometimes DevOps implementation can result in more technical debt than existed before it!

Unfortunately, quite a lot of dirty code cannot go through the refactoring or CI/CD pipelines with ease. Most often this is due to a fact that any application in production interacts with multiple entities — users, APIs, operating systems, virtual machines, cloud computing resources and so on — and any of these interactions can be impaired by fixing the technical debt. Even the automated unit testing routines can include the flawed patterns inherited from manual testing and added to CI/CD pipelines due to the business needs, even if DevOps team was against it.

As a result, the technical debt that has deep roots in the product or workflows can be extremely hard to excise. Either the company staff responsible for DevOps transition does not know about these roots, or it does not want to cut them off — in either case, the issue cannot be solved by implementing automated software testing and delivery, as the problem lies in another plane. What is even worse, the technical debt carried over from legacy infrastructure to DevOps workflows becomes even more dangerous and toxic, as it affects much more than the code now — it can potentially impair the integrity of the whole system.

Final thoughts on technical debt and its impact on DevOps

Thus said, DevOps methodology has a strong countermeasure to technical debt — and it is the dynamism and fluidity inherent to DevOps practices. Nothing is written in stone when you work according to the DevOps model, and every aspect of operations is revised from time to time to make sure its performance is optimal. Such revisions give a pretty high chance of discovering even the most deeply hidden blobs of technical debt. In addition, the constant strive to improve through implementing new approaches and processes can help find the solutions that were not available back when this technical debt was created. In theory, this will lead to a significant decrease in the volumes of technical debt with time. In real life though, there are no guarantees your product code and infrastructure are free of technical debt.

How does your company approach the task of elimination of technical debt?