How A Continuous Testing Strategy Powers Continuous Delivery

How AI-powered continuous testing empowers continuous delivery

Most *aaS companies use continuous delivery. However, it poses some challenges for testing. In this blog, we will explore these in detail and show how Functionize helps to overcome them.

‍

Continuous Delivery is increasingly seen as the norm in the world of Everything as a Service. Companies stay ahead of the competition by releasing new features as they are developed. At the same time, it helps to minimize the risks inherent in releasing new features. Gone is the idea of a red-letter release day, nowadays some companies make releases daily.

What is continuous delivery?

Continuous integration and continuous delivery (CI/CD) have been around for well over a decade. CI is the process of continually merging and building code into release candidates. Most agile companies adopt this approach, aided by tools such as Jenkins and Travis CI. CD is the process of taking that newly built code and releasing it as soon as it has passed QA. True CD implies that a new release candidate is tested and then released every single time the code changes. Over recent years, this has started to become increasingly common as a strategy.

For companies selling their products using the *aaS model, CI/CD makes a lot of sense. While we think of giants such as Amazon, Google, and Microsoft, the *aaS ecosystem is much larger. The AWS marketplace alone lists over 4,500 companies that are selling their products using the SaaS model. Clearly then, this is a highly competitive market. In turn, that means the quicker you get new features to market the better you can compete. Furthermore, by building and testing one new feature at a time, you can speed up the whole QA cycle. If only one thing has changed, you know exactly what to blame if a bug is found.

‍

What continuous delivery needs

The impact of CD on your testing strategy

CD makes some specific demands on continuous testing and on you as a tester. It impacts test planning and regression testing, as well as causing test maintenance issues.

CD speeds up the release of new features. But without a strategy of continuous testing, CD can't work. This place some significant requirements on you as a tester. Firstly, you have to develop new tests for each feature as it is added to the codebase. Thus, CD is breaking the traditional plan-test-release cycle. Secondly, every release still needs to go through regression testing. Clearly, this is only possible if you are using test automation for your regression tests. Thirdly, you now have less time to spare for essential test maintenance. This, in turn, may make your colleagues see you as a blocker to the release. Let’s look at each of these in detail.

Test planning and development

Traditionally, test planning involves you discussing the new feature set with the product team. This allows you to discover how the features work and what changes they make to the e2e flow. In turn, you are then able to develop new tests (or adapt existing tests). The aim is to create tests that touch every new function in the code. Once you get the initial versions of the new code you then modify and refine your test plan. Finally, you have a fully-functional set of tests for the new features.

Under true CI/CD you no longer have the luxury of time. So, the whole process of test planning and test development is replaced by continuous testing. Developers will still give you advance warning of what new features to expect. However, there will often be no time to develop the tests before the CI system passes the new build to you.

Regression testing

Under a CD strategy, regression testing becomes even more important than ever. Traditionally, regression testing is done between releases. You steadily work through the full suite of regression tests while the new features are tested and debugged. This is where test automation helps you out. With truly continuous delivery, you suddenly have no time to do all the tests every release. Suddenly, you have to prioritize your regression test suite based on your experience and knowledge of the product. So, we are in a world where smoke tests and regression tests are synonymous.

Test maintenance

As you well know, test maintenance can easily take up half your time. Especially, if you are working on testing UIs or the like. This is because any change to a UI can easily break all your existing automated tests. With continuous integration, there’s an increased temptation for new UI changes to be made piecemeal. This means you may even face a greater risk of spurious test failures. So, now you have to delay the release to fix the tests. And chances are at least some of your colleagues won’t be sympathetic to your plight.

‍

How autonomous testing can help

Here at Functionize, we specialize in autonomous testing. We have been developing and improving AI-powered continuous testing solutions over several years. Our aim always is to make testing easier, more intuitive and quicker, thus improving the life of testers. Many of our innovations in this area solve the issues mentioned above. So, let’s look at how we can help you.

Autonomous test creation without scripting

Last year we were proud to announce the release of NLP Engine a new approach to creating tests. ALP allows you to write a test in plain English. The system will then take this and generate a fully-functional test script to run on the Functionize Test Cloud. So, you might ask, how can this help speed up continuous delivery? Well, the answer is simple. By reducing test creation to simply writing a description of the test, we save you significant time and effort. In turn, this means you can properly plan and develop tests like you always did in the past.

We are also working on developing autonomous template recognition. This has the potential to further improve the test creation process. Effectively, it autonomously identifies and creates tests by looking for certain standard templates or functionalities.

Regression testing and canary testing

Functionize integrates with the majority of CI tools. This means that it is easy to trigger regression testing the second new code is built. Continuous testing like this isn’t unique to Functionize. However, as we will explain below, Functionize tests are also far more resilient than standard automated tests. This is thanks to features such as self-healing and one-click updates. However, even more exciting for continuous delivery is our new approach to canary testing.

Canary testing involves using real users to do your regression testing for you. All big cloud providers use canary testing for rolling out new features. Essentially, a small subset of users receives the new code and you compare their experience with existing users. Functionize has taken this one step further using a combination of AI and advanced statistical methods. We can now identify with high accuracy what a user will do next as they interact with the app. This means we can automatically compare their experience of the new codebase with equivalent users of the existing codebase. If we spot an anomaly this can be flagged or can trigger an automatic roll-back of the new code.

Intelligent test maintenance

One of the most powerful features we bring to autonomous testing is our ML Engine. This suite of features dramatically simplifies the job of test maintenance. In Functionize tests, selectors are fingerprinted rather than being treated as static objects. So, rather than looking in a specific location, a Functionize test looks for the button that achieves the required function. If the button moves or is restyled, this allows the test to self-heal.

Sometimes, a change is more fundamental, or there is confusion because multiple potential selectors exist. This can lead to test failures that are hard to spot because the root cause was several steps earlier. Functionize’s root cause analysis (RCA) uses machine learning and intelligent rule-based expert to find the likely cause of the failure. The system can test likely solutions and present these as a list of smart suggestions. These will be ranked according to how likely they were to be correct. This means you are now able to look at the list of suggestions and perform one-click updates for that test. Importantly, these updates will then be propagated to all tests that make use of that same test step.

Conclusions

Functionize is the perfect continuous testing solution for companies that need to do true continuous delivery. We believe that our unique approach to autonomous testing will help deliver better software for your users. Like all good companies, we also listen to our customers and strive to continuously improve our product line-up. Stay tuned for news of some exciting and innovative products this year.