AI test automation

What it is, when to use it, and how it can help your business

AI is transforming business and AI test automation can bring you real benefits. We look at what AI is, how it applies to test automation, and how this can impact on your bottom line.

Introduction

There can be no doubt that AI is transforming businesses across every sector from apps to automobiles and healthcare to haulage. But, as with all buzzwords, there is a lot of hype round AI, much of which is premature. In this blog we try to take a more considered look at AI test automation. We present some strategies and show how autonomous testing can transform UI test automation.

Artificial Intelligence

A short overview

By now, everyone has heard about AI. But AI is nearly as broad a term as IT, so let’s take a quick look at what it means. It’s important to point out that AI covers a number of different fields. Even within a particular field of AI, there are usually several different approaches and technologies. But common to all of them is the idea of taking some data and using intelligence to extract patterns or meaning from it. Here we only have space to give a brief overview, but there are plenty of resources out there if you want to learn more.

Machine Learning

Machine Learning is probably what most people think of when they think of artificial intelligence. ML is all about training a computer model to recognize certain patterns in data. To do this you need 2 (large) sets of known data – that is, data that is already classified correctly. The first set is used to train the algorithm. Once it is well-enough trained, the second set is used to test it. Finally, it is ready to categorize new, unknown data.

Deep Learning

Strictly speaking, deep learning is a form of machine learning. Typically, it uses deeply-nested artificial neural networks (ANNs), hence the “deep”. The key difference here is that deep learning enables unsupervised and semi-supervised learning. In other words, you don’t need a fully-labeled dataset to train the algorithm. Instead, it learns by its mistakes (much as we do).

Image recognition

Image recognition, or computer vision, involves teaching computers to recognize still and moving images. Often this is used with a special class of neural network called a convolutional neural network. There are two main uses for this. Either you are looking to either identify a specific object that may be in the image. Or you are looking to classify all the objects in the image.

NLP

Natural language processing is the process of teaching a computer to actually understand human language. It is a rapidly growing field and is a key technology behind virtual assistants such as Amazon Alexa. (N.B. these also rely on other AI technologies such as speech-to-text and text-to-speech).

Generative Adversarial Networks

This is quite a new form of AI. The aim is to teach computers to create new images or videos using two different neural networks. The first one takes random noise and uses this to create new images. The second one compares the new image against a training set of images. The aim is for the new image to be indistinguishable from the training set. This allows AI to create paintings, such as the one that sold for $432,500 at Christie's last year.

What is AI test automation?

Any successful test automation strategy relies on three key things. Firstly, you need to be able to create new test scripts. Secondly, you need to be able to analyze and identify test failures. Thirdly, you need to be able to maintain your tests as your product evolves. All three of these can be improved using artificial intelligence. Let’s explore these in order of complexity.

Identifying and analyzing test failures

There are a couple of different ways that AI can be used to identify test failures. The “dumbest” way is to use image recognition to spot when something is wrong on the screen. This can be dumb, where any change is flagged, or it can be more nuanced like Functionize’s fuzzy image analysis. The other way to spot issues is to use machine learning to understand the actual content of the UI. Here at Functionize, we make use of both these approaches.

Test maintenance

One of the banes of test automation is that even the slightest change in the UI can trigger spurious test failures. Traditionally, this meant every UI change required all the test scripts to be updated. Using a combination of deep learning and other techniques, it is possible to train an AI to identify these cases. Here at Functionize, we have various approaches including our Root Cause Analysis, which tracks test failures back to the actual trigger (which is often several steps prior to the test failing).

Test Creation

One of the hardest aspects of test automation is creating test scripts in the first place. It is this that has led to the rise of the “Developer in test” and other such job titles. AI can help you create tests using an intelligent modeler (like Functionize used to offer) or using NLP. This second option is now our preferred approach because it removes all barriers to test creation. Anyone that can describe a test can now create tests.

Should I be doing test automation at all?

Exactly how you can use AI test automation will depend on the nature of your business. Let’s look at some of the factors you should be considering. First, we’ll look at whether you should be doing test automation at all

Type of product

Not every product is equally suited to test automation. Generally, test automation is best used for regression and functional testing of user interfaces. If your product only has a simple user interface, it may not be worth investing time and effort into automating the testing. OTOH, a complex UI will really benefit from test automation, but this will be harder to achieve.

Development strategy

If you are doing CI/CD, you will definitely benefit from test automation. Indeed, without this, you will find it almost impossible to regression test new features. But if you are only releasing once per year you will have plenty of time to test more manually if you need/want.

When will AI test automation help me?

AI test automation will be a huge benefit if you have a very dynamic product with a complex UI. If you are pushing new features regularly and find your testing team is spending a lot of time on test maintenance, you should definitely consider adopting an AI system. We often hear of test automation engineers spending over half their time on maintenance. That is time that could be used to automate more tests. So, there is also a secondary cost in needing to do more manual testing. This was highlighted in our recent Case Study with TOTVS.

It can also help if you are short of test automation engineers, or if your engineers cost you a lot. As you probably know, a test automation engineer can earn well in excess of $100k. That’s about 20% more than a normal QA engineer. Developers in test can earn even more. Systems such as Functionize’s Adaptive Language Processing™ mean anyone can create tests. And our Adaptive Event Analysis™ allows even unskilled test engineers to perform test maintenance. This means you can automate more tests, leaving your team free to concentrate on manual and exploratory testing, or even more strategic activities.

Are there other ways an AI test automation strategy can help?

AI can’t achieve miracles, but it can sometimes seem miraculous. Up till now, we’ve been talking about systems that test HTML-based UIs. That is websites, web apps and the like. But what if your system is based on an old or native UI? Well, AI-powered test automation may be your savior.

Using a combination of AI methods, Functionize has created an approach we call autonomous template recognition. For each UI screen, the system defines a dynamic template listing the items that should appear. This includes metadata such as the relative position of icons and text on the screen. 

The upshot of this is that our system can autonomously locate the home screen on the system under test, then proceed to navigate around the system by matching icons against labels and performing suitable actions. The system can replicate numerous human-computer interactions such as clicking, dragging, scrolling, button presses, etc. As the system performs the tests it is constantly recording the screenshots, checking for any visual anomalies and creating warnings unless these are in dynamic data.

Conclusions

Not every company will benefit from AI test automation. But many companies will, and it can lead to significant cost savings. It can also dramatically boost productivity. Clearly, the specific solution you choose to go with will depend on your own circumstances. However, we encourage you to be slightly skeptical about some of the claims for AI. Ask any potential supplier to explain exactly what their system is really doing, then ask yourself if it’s really intelligent. Here at Functionize, we feel confident that we are leading the field in autonomous testing.