Continuous testing plus DevOps equals DevTestOps

How continuous testing enables DevTestOps

Could this be the next revolution in software engineering?

DevOps blurs the line between developers and operations, allowing companies to streamline their product releases. When you add continuous testing to the mix, you accelerate this process even more. The end result is DevTestOps, a new philosophy for agile teams.

Background

Everyone has heard of DevOps, and most companies have started to embrace it as an approach. DevOps has a huge impact on software deployment, failure recovery, and feature delivery. But DevOps can only take you so far. Enter DevTestOps, where the worlds of DevOps and Continuous Testing merge. In this blog, we’ll look briefly at DevOps, explore continuous testing and introduce you to the DevTestOps Manifesto. By the end, we hope you’ll see why this approach promises to be the next revolution in software development.

‍

DevOps

A much (ab)used term, but what actually is it?

DevOps, a combination of development and operations, was coined almost a decade ago. The idea of DevOps is to try and reduce or remove the gap between developers and operations engineers. The aim is to streamline the process of releasing new features.

According to Amazon: 

“DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.” 

Sometimes, there is confusion between DevOps and Google-style Site Reliability Engineering. But Google makes it clearer, explaining that “Class SRE implements DevOps”. In other words, SRE is just one of many approaches that achieves the aims of the DevOps philosophy. My advice would be that most companies can’t afford to take the SRE approach. But that doesn’t stop you from adopting the DevOps approach.

Continuous Testing

Recently we looked at the related concepts of Continuous Integration and Continuous Delivery. Continuous Testing takes this philosophy and shifts it left. The key aim of continuous testing is to bridge the gap between developers and testers. Historically, developers would complete their work, then pass the code to testers. If there were errors these were passed back. Else, the code was released. There were always exceptions to this. For instance, the developers always did unit testing themselves. But as a rule, the two roles were completely separate.

Over time there has been a move to shift testing left, meaning moving it earlier in the development cycle. Some development methodologies embrace this completely, such as Test-Driven Development. Key to the shift-left process is to blur the boundary between developer and tester. As a minimum, testers should be embedded into developer teams. And developers should be given the power to run tests themselves to speed the whole release cycle up. 

Continuous testing is the logical endpoint of this process. The aim is to provide continuous feedback to the developers about the state of their code. This should expose any problems early on. This reduces the business risks (we all know examples of features taking months to develop before failing dismally in testing).

Requirements and challenges

Test automation sits at the heart of continuous testing. Test automation allows anyone on the team to run tests and analyze the results, whenever needed. This implies testing should be happening at least daily. So, this means tests should be:

• Deterministic, to make analysis easy.
• Modular, so a developer can test just her part of the system.
• Compartmentalized, so that issues in one test don’t impact other tests.
• Repeatable, so that results can be easily compared.

Also, in order to give the most value, tests should be run in production-like environments. This makes continuous testing potentially costly to the business, at least in terms of cash. So, it’s vital to highlight the benefits in terms of reduced time to market.

There are a couple of challenges for continuous testing. Firstly, it’s hard to automate the testing of new features. This is because you can’t develop the test until the feature exists. Secondly, test automation is often brittle, especially when code is changing frequently. This means test automation implies significant test maintenance. Thirdly, by definition in any modern distributed app there will be many dependencies. This makes it hard to achieve modular testing. The solution is to use things such as mock API endpoints. 

The V model for testing. In continuous testing, all stages of testing happen earlier.

DevOpsTesting

“The goal of DevTestOps is not to silo "Test" from Dev and Ops, but simply to raise the visibility of testers and testing as integral parts of the DevOps quality culture.” (DevTestOpsManifesto, 2018)

DevOpsTesting is the culmination of all the elements above. DevOpsTesting blurs the lines between developers, testers and operations engineers. According to the people behind the DevOpsTesting manifesto:

“DevOps evolved from agile development’s whole team approach to building quality in. Testers and QA professionals are part of cross-functional software delivery teams that practice DevOps.” They make the key point that testing should be part of every discussion and conversation. “…from building shared understanding as new features are planned, to managing the delivery pipeline, to monitoring and observing production usage, to shortening the mean time to recover from production issues”.

This led to the actual manifesto, which pays homage to the Agile manifesto:

• Continuous testing over testing at the end
• Embracing all testing activities over automating everything
• Testing what gives value, based on customer usage, over testing everything
• A whole team approach to testing over testing in siloed testing departments
• Product coverage over code coverage

There are some perhaps-surprising elements to the manifesto. For instance, they highlight that DevTestOps requires you to embrace all testing, not just test automation. Thus, exploratory testing, test development and manual testing are all part of DevTestOps. They also make it clear that it’s not just about running every test. What matters is to test the things that matter to the product and the customer.

Implementing DevTestOps

Let's look at how you might implement this manifesto. For starters, you need to have the right systems and procedures in place. This means you need a proper test automation system. You need to embed your testers in the development team and avoid the old-fashioned silos. You should also be prepared to embrace the DevOps philosophy as a team.

Focusing on the product requires collaboration with the product team. You should work to prioritize what matters to your customers. One of the issues I often see is a mismatch between what developers think is cool and what customers actually want. At its worst, this leads to feature bloat. And certainly, it’s important not to expend effort on testing features that are not being used. A feature may seem cool, but if no customer ever uses it, there’s no point expending effort testing it. 

Full implementation of DevTestOps will need buy-in from everyone. Management has to get on board and understand that increased costs are far outweighed by increased efficiency. Product managers have to work with everyone to ensure they communicate customer needs. Developers have to adapt to a new way of working, where they are responsible for testing their code. Testers become a source of expertise, helping the team to develop suitable tests, efficiently. The operations team now have to look after the development and staging environments, making sure there are no mismatches. Finally, the whole team has to accept that testing happens at every stage in the process from inception to production.

How Functionize helps

Functionize specializes in autonomous testing solutions. Our whole philosophy is to remove the pain points of test automation by using artificial intelligence. As a result, we think our system is ideal for implementing continuous testing and, by extension, DevTestOps. Three features, in particular, will help. These are NLP Engine, ML Engine and autonomous canary testing.

NLP Engine is our test generation engine. Tests are written in plain English and the system converts them into fully functioning test scripts. The huge benefit here is that you can define the test even before the product is built. Even better, it doesn't require a test automation expert to create test scripts. Both developers and product managers are now able to define fully-functional tests. 

ML engine is our autonomous test maintenance and repair system. It is designed to reduce or even remove the pain of test maintenance. This is essential if you are going to adopt continuous testing. It consists of three main parts:

1. Self-healing tests, where the system uses machine learning (ML) to automatically recognize changes to layout, style, etc.
2. Root cause analysis, which uses ML and an advanced rules engine to establish the real cause of a failure.
3. Smart suggestions and one-click updates, where the system works out the best solution to the failure. You simply click to accept the correct solution.

Autonomous Canary Testing allows you to automatically test new code in production. Around 5% of your users receive the new code. The system compares their experience with that of customers on the old code. Any discrepancy is flagged and can trigger a rollback. If the code is stable, you can automatically deploy it.

To find out more about these features check our website and blogs, or speak to one of our solutions architects.