Must have DevOps tools: make things right from the get-go

While DevOps is not purely a set of tools and practices and rather is a mindset and an approach to software delivery, using the must-have DevOps tools makes the workflow much better.
The DevOps workflow consists of several phases that require certain features from the DevOps tools used. The list of the phases themselves includes the following:

• planning the next iteration of the product development,
• building the code to release the next batch of features,
• testing and deploying them to the production environment automatically,
• delivering the product updates without interrupting the end-user experience,
• monitoring and logging the software performance,
• gathering the customer feedback and planning the next software development sprint based on this feedback.

It is clear these phases involve different tooling for different companies. However, there are some best practices, which ensure getting the best results. These practices involve using the top DevOps technologies, either open source or proprietary. The tech in question is available from popular Cloud Service Providers like AWS or Azure, Google Cloud or DigitalOcean. Below we list the DevOps software in question and explain why we consider some of the listed DevOps software to be the top performer in its category.

Iterative planning

As DevOps workflow is a natural evolution of Agile software development paradigm, it suggests delivering new software features in rather small chunks of code created during short sprints. This means a schedule-planning and task-tracking tool is needed in order to ensure the team knows what tasks are at hand, what is currently being done and if there are any risks of falling behind the schedule. Tools like Confluence, Jira, Redmine, and Trello help DevOps teams achieve seamless and efficient project management cycle and ensure timely product delivery.

Main benefits of project tracking systems

The tools listed above help visualize the progress of project delivery and provide other benefits:

• Centralization of tasks, issues, and reports
• Accountability, as the tasks are easily assigned and checked
• Granular control over permissions, to ensure every user sees only what he needs to see
• Logging and history, allowing to track any change and determine the root of the problem
• Notifications, to ensure all team members are updated on the important events

Here is why Jira is in the list of top issue tracking tools for DevOps engineers:

• Sprint planning, task delegation, user stories and issue creation
• All the work done is visible and can be easily discussed
• The latest code batches and software features are always available
• In-depth statistics to keep the hand on the pulse of project progress
• Multiple basic workflow models or custom workflows to maximize the team performance

Building the code

Developers need rapid deployment of development and testing environments and don’t want them repaired for long when something goes wrong. Leveraging Docker containers ensures delivering pre-configured software ecosystems as many times as needed, given they work the same on any machine that runs them, so “works on my machine!” is no longer an issue. This alone is a blessing for software developers.

To say even more, implementing the Infrastructure as Code principle, when deployment of a new environment is easier than repairing the broken one — this takes the software development workflow to a whole new level. The list of DevOps tools for this phase include Kubernetes, Terraform, Chef, Ansible, and Puppet, among others.

Main benefits of infrastructure management systems

Using configuration management tools like Kubernetes or Ansible provides immense benefits:

• Efficiency, through using “infrastructure as code” principle, as it ensures significant speed-up of the infrastructure configuration, as compared with the manual setup process
• Auditability, meaning the configuration management tools perform the constant check of operating conditions of the infrastructure, to ensure the app works as intended
• Removing the configuration drift, as the same changes are made to all managed environments at once to ensure there are no unique system setups — and faults
• Documentation blueprint, as Kubernetes or Terraform manifests already contain a ton of information on the desired infrastructure state, so there is no need to describe it in the text in whole. Using the CM tool removes a huge amount of effort from the DevOps team and developers in terms of documenting their actions and configuration model
• Effort consolidation, as the built-in system logs keep track on every update and change done to the system, so all the effort spent is duly noted. The new team members can get up to speed quickly, as the tool helps to understand the existing infrastructure in an easy-to-grasp, standardized way

Here is how Kubernetes simplifies the infrastructure management:

• Cloud-agnostic layer for Docker container management to avoid vendor lock-in
• Pods provide modularity, scalability, load balancing, and ease of monitoring
• Kubernetes Controllers enable rolling updates and canary deployments
• Actively developed open source framework for delivering production-ready environments

Automated testing, continuous integration, and delivery

When the code is written, it must be tested to ensure everything works as intended. As manual testing becomes the thing from the past, automated unit testing is the mainframe approach to software delivery. Once the unit tests are written, the testing and continuous integration / continuous delivery (CI/CD) pipeline can be set, where the testing environments are provisioned automatically, the code is tested and pushed to production if everything went well (or the process is halted and a smart alert is raised if there were any issues).

Such workflow helps minimize the time and effort devoted to testing without compromising the code quality or user experience. We recommend Jenkins, CircleCI or Gitlab CI for that matter. This also helps standardize the update process and avoid configuration drift, where different instances have different product versions, resulting in unneeded admin and management overhead. The latest stable version of Jenkins provides unmatched capabilities for building CI/CD pipelines using Jenkins Workers. In addition, they provide native support for AWS EC2, allowing to save the costs significantly through the use of Amazon EC2 Fleet Plugin.

Main benefits of CI/CD tools

Jenkins and other CI/CD tools provide the following advantages for their users:

• Bug risk reduction, as automated testing of the code according to predefined production environment conditions means the bugs are detected much earlier along the software delivery pipeline
• Confidence, both in your product and in your team, as successful completion of tests means there are no bugs shipped in this release and the update will be successful ( and if something does go awry, there is a backup and an automated rollback procedure)
• Improved communication, as the tools like Jenkins can generate smart alerts when an important event occurs, and send them to various messengers like Slack, Telegram or WhatsApp. Your team will always be updated if the build was successful or if any issues should be solved before pushing the code to production
• Decreased operational overhead, meaning lots of routine actions can be automated, including testing. This way, the time will be spent generating value, not doing repetitive monkey work, not to mention new team members will be able to onboard faster if they will not have to learn in full what the system does in the background
• Build process consistency, which means that pre-configuring the build process instead of doing it manually allows creating the same conditions each time, independently of IDE’s used. “Works on my machine” is no longer an issue and not a single step will be overlooked by the CI/CD tool

We consider Jenkins the most efficient tool of the three listed above, for the following reasons:

• Standalone Java app, ready to run out-of-the-box on Windows, MacOS or Linux
• Simple and elegant CI/CD pipelines creation and management
• Simple setup and configuration using a web interface
• Literally hundreds of plugins available to customize the app in the Update Center

Software monitoring and logging

Once the updated software version is moved to production it must be monitored to ensure stable performance and increased customer satisfaction. This stage also involves performance analysis and logging, raising smart alerts on various issues, gathering customer feedback on the new features, etc. There is a plenitude of tools for performing these tasks, most prominent being Prometheus, Grafana, ELK Stack, Splunk, Sumo Logic, etc.

Main benefits of using software monitoring and logging tools

The benefits of leveraging specialized tools for monitoring and logging the apps in production are obvious:

• Identifying the issues before they occur, as network monitoring allows understanding the current state of affairs within the infrastructure and if there is trouble in the making. The tools can monitor the parameters like CPU or RAM usage, disk volume usage, I/O throughput, etc. — and warn the support team once the values exceed certain thresholds
• Solving the issues faster, as when the app or website downtime does occur, lost time literally becomes lost money. Multiple built-in features and additional plugins for popular monitoring and logging tools allow turning disparate data and statistics into quickly understandable graphs. Using these features along with various data visualization tools ensures the time needed to assess the situation is minimal and the team can concentrate on solving the issues, not on looking for their roots
• Gaining ROI from the start, as the infrastructure support team can minimize the time spent troubleshooting, thus reducing the downtime (and the risk of downtime, in best-case scenarios). The freed time can be devoted to improving the infrastructure and delivering more value, instead of putting out the fires permanently
• Manageability, as the environments monitored are constantly growing in size and complexity. Gartner reports there will be more than 20 billion interconnected devices by 2020, and at least 3 billion of these will form corporate networks. Every business must make sure they have robust and reliable cloud monitoring solutions in place in order to cope with the ever-growing complexity of projects and networks
• Security, as by constantly monitoring the infrastructure performance, the cloud engineers are able to identify and describe the normal patterns of usage. This, in turn, enables them to highlight the abnormal behavior like unidentified device trying to access the network or sudden rise in the traffic and server workload. Monitoring tools can be efficiently used as the first line of defense for a multi-layer enterprise quality security suite

IT Svit considers Prometheus+Grafana to be the most efficient monitoring software, and here is why:

• Versatile query language to process the collected data and build the needed graphs
• Built-in data scaling and federation to ensure stable data storage
• Works with more than 10 coding languages and supports multiple custom libraries
• Smart alert manager for configuring and managing alerts

Gathering the customer feedback is essential to understand the desired way to evolve the product, yet it is most often done not by DevOps team, and not using the DevOps toolkit. Thus said, this important part of software delivery lifecycle will be left outside of the scope of this article.

Final thoughts on must-have DevOps tools for any team

Each business has a unique operational DNA and might use a varying set of DevOps tools. Your company can house a rather small DevOps team responsible for the full product delivery cycle, or quite a few teams working on separate projects. In either case, each software development team can benefit from organizing their workflow according to the best DevOps practices.

The tools you choose for each process stage might vary, as it’s the collaborative approach and bolstering the DevOps culture that really matters. In the end, even the must-have DevOps tools you use do not matter much, as long as the team shares the responsibilities, skills, and attitude to improve the DevOps software usage in the product delivery workflow. Thus said, the most important of the DevOps instruments are still git, IDE, and your own brain 🙂