Testing in a DevOps Mindset
Do your development teams follow DevOps, or are you considering implementing a DevOps culture in the future? If so, you need to know how to approach software testing with a DevOps mindset.
AWS defines DevOps as follows:
"...the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes."
A DevOps adoption aims to unify development and operations, with a strong focus on automation and application performance monitoring at all steps of the development pipeline.
The need for DevOps arose from Agile practices such as application automation were used to release software faster in Agile teams, but the flood of smaller more frequent releases strained QA and operations teams. Then came the ideas of treating the entire software lifecycle as a single process, automating that cycle, and, therefore, reducing downstream pressure on operations. The end cultural movement was a set of practices that view software delivery as a strategic business initiative.
Testing software in DevOps is similar to Agile in that testing is conducted earlier, at all stages of development. DevOps presents its own unique testing focuses, challenges, and test metrics for improving testing processes in future releases.
DevOps Testing Processes and Best Practices
Test Automation Frameworks
One thing you cannot avoid in a DevOps culture is using test automation frameworks. Since automation is at the core of DevOps and one of the key drivers of DevOps objectives, you must use a mature automation framework that can effortlessly create scripts for new test cases.
A framework is a set of best practices, assumptions, common tools, and libraries that can be used across teams.
You are faced with the choice of custom building a framework for your specific team's needs, using an open source automation framework, or buying a commercial framework. One of the latter options is more practical because there is no formal support and if the creator of a custom-built framework leaves the company, you might be left not knowing how to use it.
Serenity and WATIR are two examples of open source automation frameworks. Commercial options include Tricentis Tosca Testsuite and Smartbeat TestComplete.
Even though automation frameworks might cause a slowdown in the speed of releases as they are first implemented, they always deliver value in the long-term.
Automation frameworks help prevent defects instead of focusing on finding them, and they help teams create test suites much quicker by abstracting away UI details with reusable blocks of common functionality, which makes tests easier to
Counters that automatically monitor response times, CPU utilization, and memory usage can provide insights into potential defects. For example, if a certain process seems to take longer with each build, such as logging into a system, future builds might become unusable due to high response times.
Application Monitoring
Early detection of software defects before they cause issues such as system failures is a vital aspect of DevOps testing. With this in mind, application monitoring tools are crucial and must be deployed in the production environment to expose bugs before they cause a crash.
Prometheus is a popular open source monitoring tool that enables you to handle a large amount of metric data. The Prometheus query language enables you to visualize data and manage alerts. Prometheus supports many different integrations, including with PageDuty and Slack for alert notifications, and with Grafana for visual dashboards. The list of supported products include server applications, database products, Kubernetes, and Java Virtual Machines (JVM).
Test Orchestration
Test orchestration is the natural result of the DevOps focus on automation. Your team ends up running your entire software test workflow in repeatable, reusable steps that can be optimized to release software quicker. DevOps teams often use orchestration tools for software testing efforts, such as Cucumber.
The subtle difference between automation and orchestration is that automation is concerned with single tasks, such as automatically stopping a service, and orchestration is for automating and optimizing a workflow, or process, involving multiple tasks and multiple systems. Orchestration deals with the big picture, considering how particular decisions affect the entire testing process.
By ensuring a testing process is repeatable and orchestrated, development teams optimize the testing process by removing redundancies, which improves the speed at which you can deploy software.
Test Maturity
Test maturity is a key differentiator of software tests for development teams following DevOps. When testing is not fine-tuned, improved with each release, and streamlined,bottleneck issues for software deployment arise. Automation, and more importantly, orchestration, is what makes the difference to testing maturity. At the mature testing level, teams optimize their testing processes for faster releases.
Test Metrics
Naturally leading on from the last point is the challenge of improving testing processes all the time, what is it that leads to such improvements? As you've found out, orchestration and optimization is key. But there is another important factor—test metrics.
Test metrics are still relevant in a DevOps environment as with any other software development approach, and the right test metrics can further streamline testing processes by providing valuable insights on testing efforts and software quality.
Some examples of test metrics relevant in a DevOps context are:
- Automation Progress—since test automation is an important driver of the reduced deployment times that DevOps aims for, it makes sense to track your automation progress over time. Automation progress is a measure of the proportion of test cases automated against the total possible automatable tests. Ideally, you want to see positive progress as soon as you start practicing DevOps.
• Deployment Frequency—this a simple yet insightful metric that provides a clear evaluation of whether your dev teams are deploying new code frequently enough. The metric should trend up over time or remain stable—any decrease in deployments warrants investigation into possible bottlenecks.
• MTTR—mean time to recovery measures how long it takes to recover from a production failure. This is a pivotal metric in understanding testing maturity in DevOps—with automated unit tests, acceptance tests, application performance monitoring, and test orchestration, issues should be found and fixed quickly, within minutes. MTTR must be kept to a minimum, and should decrease over time.
• Test coverage—while test coverage doesn't tell you anything about the quality of tests, it still serves as an important indicator of how likely you are to catch critical or high priority bugs. For example, code coverage is just one of many useful code quality metrics that can be complemented by other useful test metrics, including requirements coverage, which measures how many test cases are written that cover customer requirements.
Closing Thoughts
- There are several tenets to testing software in a DevOps mindset, each of which is used to help achieve the set of goals aligned with a DevOps initiative—standardized development, automated delivery, faster time to market, and a higher frequency of deployment.
- DevOps testing has a heavy focus on automation—both in automating single tasks or tests, and streamlining and optimizing the entire testing process, through test orchestration.
- Automation frameworks, test maturity, and application performance monitoring help further optimize software testing in DevOps.
- Finally, test metrics are still important in teams following a DevOps culture because tracking the right metrics leads to improvements and refinements in testing.
