Best practices of implementing DevOps in the banking industry

The banking industry has invested billions of dollars across the last 2-3 decades into their IT infrastructure in order to be able to support the rapidly-changing preferences of their customers.

Mobile apps for the younger generations like Millennials or Gen Z, web apps for Gen X, who are mostly not into mobile apps, live support centers for Baby Boomers with 2nd and 3rd line of admins dealing with mainframe infrastructure underneath — all of this has to work coherently and interact flawlessly with each other. Now, it is time for adopting DevOps in the banking industry, in order to be prepared for the future. Is your bank ready?

Current problems with software development in the banking industry

As of today, most of the retail and investment banks and other financial services companies have a somewhat outdated approach to the development and operations of their software. The banks must be secure, and in many cases, this is achieved through strict control of the software development lifecycle and working according to the Waterfall software delivery model. This includes rigid technical requirements, lengthy development process with multiple approvals between stages, long-term testing and infrequent product updates — everything to emphasize operational stability and security, yes?

However, most of the banks witness the direct opposite outcome:

Legacy infrastructure and applications are extremely complex systems, composed of a mix of manually-configured “snowflake servers”, as well as a tangle of proprietary and custom software held together by an inordinate amount of scripting. Few banks are lucky enough to have some of the initial system developers in place, who know how the system actually works. But mostly, people tend to move on to greener pastures and more agile software development processes, and the bank IT departments are left with in-depth textbooks manuals and a mess of legacy code and scripts they have to make work somehow.
Legacy infrastructure means legacy tools, as people have to work with outdated instruments to support outdated servers. This slows both the software delivery and operations, as most operations are either manual and based on working with GUI, or scripted routines using bash and Powershell scripts.
Waterfall means the updates are delivered after a long while once requested, so they are usually out of sync with the actual needs of the moment, as the requirements change over time. In addition, there are several “freeze periods” like Thanksgiving and Christmas and summer vacation season, when no updates are released to ensure smooth banking experience for the customers.
Approvals end up merely slowing the development process with little positive outcomes, as the requested changes have to be approved no matter what in the end (as there is the point in software delivery otherwise) — so why bother wasting time on multiple approval levels at all?
Test-driven development is great if it is done right and from the very start. The issue with the testing process in banking industry is that the production systems are very diversified and testing servers are not identical to production ones. Thus said, the differences between the testing and production environments result in release-day crashes, our next point of the challenge.
It is normal for the bank IT departments to shut down their systems on weekends and try to push releases to production overnight, like from Saturday midnight to Sunday morning, when the workload is minimal. However, as we mentioned earlier, border cases unaccounted for during the testing can lead to crashes in production environments. This means that the backups must be made prior to updates, and the changes have to quite often be rolled back to fix the newly-found bugs, thus slowing the release for another week.

Sounds like a nightmare, eh? Or is it a normal operational routine in your bank? However, banks and financial companies actually have some very strong points that can be their stepping stones to DevOps workflows and Continuous Delivery best practices.

Prerequisites for DevOps adoption in banks

There are several important traits of the software delivery lifecycle that can make the DevOps adoption in banking industry much easier:

Collaboration between developers and software engineers. The banks operate long-term projects and the teams that work there generally work together for extended periods. Thus said, the collaboration between the Devs and the Ops is a very positive outcome, as it ensures both parties understand the needs of future product architecture, its operational limits and the best way to deliver it in time, under clear KPIs and with similar incentives. This helps plan in advance and ease up the tension, instead of just the Devs throwing the code over the wall to become the problem for Ops.
Developers are like proprietors to their code. When you work on the same app for months and years, it’s better to do your job correctly right from the start, because it will be you who will be dealing with technical debt further down the line, Thus said, developers have the incentive to collaborate with Ops and ensure their code is clean and runs well, to minimize the amount of bug fixing in the future. This also means the developers have to understand the way the production environment ticks in order to be able to deal with incidents better and fix bugs faster.
Status equality between Devs and Ops. Unlike in software delivery or eCommerce or marketing or any other industry, DevOps in banking is grounded on status equality. Developers have to deliver new features on time, and Ops have to ensure stable operations of production environments so both parties can have their say when setting up the goals and aligning the business strategy and project requirements with their capabilities. This helps improve value delivery and productiveness.
Full-stack software development expertise. As the developers have to improve the whole bulk of banking systems and application, they should have a decent understanding of how the frontend, back-end and databases work in their apps, and how to manage all parts of the stack. It is quite the opposite for the practice of specialization, where front-end developers don’t care how the app will interact with the databases — it is for backend developers and DBAs to figure out, yeah?
Modular application architecture. Most of the banking industry IT infrastructure is composed of a mix of various proprietary and custom-tailored software modules that interact with the mainframe through various middleware like Informatica, Websphere MQ or Tibco. The same concept of splitting the monolith app into a group of separate microservices that interact through message brokers allows to speed up the development of the systems immensely, as each part can be developed, tested and updated separately.
Strong support of automation. As we mentioned before, scripting is the only sane way to keep these Leviathans of banking infrastructure operational. Therefore, even if the Ops engineers have to operate their own codebase of scripts using outdated tools, they do have the understanding of scripting best practices and can master CLI commands for DevOps tools like Kubernetes, Jenkins, Ansible and Terraform quit easily.
Scrum readiness across the board. While the goals are usually set according to the Waterfall model, they are delivered through a long timeline and in smaller chunks. In addition, reiterating the new code to fix bugs found during the last release actually leads to understanding the value of shorter development cycles and constant reassessment of new code — the cornerstones of Agile and Scrum.

Thus said, even if the IT departments in banks operate in an outdated way, they have all the prerequisites to undergo the successful digital transformation. It can be safely assumed that by now every bank understands the need to move from an unpredictable and highly volatile legacy software development routines and systems to a more stable, reliable and cost-efficient mode of operations. In other words, they need to move to DevOps.

How DevOps for banking makes things different

The DevOps best practices and founding principles are the culture of collaboration between cross-functional teams of Devs and Ops engineers, working with virtualized servers to operate the Infrastructure as Code, developing the code in short batches to provide Continuous Integration of feedback and use the automation tools to ensure Continuous Delivery of new code to production without disrupting its normal operation.

As you can see, this process has much in common with the best practices and procedures that are already in place in the banking industry. Therefore, all the bank or financial company needs is the external help with organizing the digital transformation and teaching their teams to work according to DevOps best practices. This help might come in two forms: ordering DevOps-as-a-Service from a Managed Services Provider (MSP), or hiring their DevOps experts to build an internal Center of Excellence. We cover each DevOps approach with their flaws and benefits below.

DevOps explained by a transition project

If a company wants to get the things done, an external DevOps team can step in and deliver the required DevOps expertise.

They assess the existing infrastructure, workflows and processes, tools and software delivery workflows in place. They issue recommendations regarding the ways to reorganize the existing systems in order to make them more simple and transparent and increase the reliability and predictability of the software delivery lifecycle.
These DevOps engineers will then create a roadmap with milestones of such a transition and help decouple the existing infrastructure, moving the system components to the cloud platform of your choice through lift-and-shift if possible, or rebuilding them from scratch with cloud-based analogs if required. After such a cloud transition, your developers and operations engineers will be able to utilize infrastructure as code, updating and managing it in bulk through manifests stored at your Version Control systems.
These DevOps specialists will help analyze the existing systems and split the monolithic apps into microservices. This will help greatly reduce the complexity of software development and updates, as well as application management in general.
They will also help ensure the compliance and security of new systems by introducing intensive security and compliance checks to the codebase of automated unit testing to exclude the human error factor from the equation. Thus said, instead of dealing with security issues right before the release, your developers perform all the required checks each time they submit a new batch of code.
Finally, these DevOps experts will configure the open-source DevOps tools and establish the required DevOps pipelines to ensure Continuous integration and Continuous Delivery of new code. This way your product will be developed in minor chunks with frequent releases that are pushed to production seamlessly through the rolling updates, with your customers even unaware of the update process — they will simply be always using the latest version of your products.

Thus said, your internal IT department will receive the ready cloud-based system with configured CI/CD pipelines, fault tolerance, auto-scaling and transparent monitoring options, along with all the developer documentation required to run it. In the process of such a DevOps adoption, they will learn to work according to DevOps culture, becoming more productive and spending fewer resources on repetitive actions.

Seems too good to be true? What are the fallbacks of this DevOps approach?

This contradicts the inherent security doctrine of every bank. Outsiders must not be given access to the sacrosanct parts of the system — customer data and financial transactions records, and these are an essential part of the transition process.
The loss of managerial involvement. Once the infrastructure is cloud-based and virtualized, new testing and staging environments can be provisioned, used and destroyed in a matter of seconds. This means the developers and operations teams will not have to wait for multiple approvals before moving to the next sprint stages. This means a huge improvement in the project management efficiency — but a huge loss of control for the managerial body.

Thus said, this does not mean the directors of banks and financial organizations will not be able to execute their control over the software development process. It only means there will be as little choke points along the software delivery pipeline, as possible.

DevOps adoption through a Center of Excellence

As most of the banks are not yet ready to entrust their digital transformation to an IT outsourcing company, no matter how trustworthy and reputable, the second way to perform a transition to DevOps is through training an internal team.

In this case, the roadmap is the same, but there is an initial stage, where the external DevOps specialists help the in-house IT department to master the DevOps tools, workflows and best practices. This takes longer, costs more and will require running several pilots, but will help form a nucleus of DevOps culture within the banks or financial organizations, which will serve as a group of visionaries and facilitators, helping the rest of the team to make the transition.

This will lead to de-siloing of responsibilities and forming cross-functional teams without letting the outsiders close to your mission-critical infrastructure.

There are certain drawbacks of this approach, however:

without an in-depth understanding of the cloud infrastructure capabilities, the aspiring DevOps engineers might design and implement the sub-par architectures unable to utilize the cloud platform capacities to their fullest extent
without an extensive understanding of DevOps best practices and possibilities of DevOps tools, as well as ready solutions for various use cases, your in-house DevOps team will have to work through the way of trial and error, which will result in a much longer transition process.

Conclusions: DevOps benefits for the banking industry

Whichever approach your bank selects, you will be able to perform a DevOps transformation of your banking infrastructure. Therefore, you can either use the help of a trustworthy MSP chosen among the leaders of the IT outsourcing industry based on the replies of their customers to deliver the whole digital transformation project or use their expertise only to provide the initial training your in-house specialists to do the job on their own.