7 ways to increase your QA coverage

QA coverage is one of the key metrics for measuring the performance of your QA team. But the complexity of modern applications makes it increasingly hard to achieve good coverage. In this blog, we will look at 7 ways you can increase QA coverage using AI tools for software testing.

What is QA coverage?

Coverage is one of the standard metrics for measuring QA teams. QA coverage measures the proportion of your software that is being tested. For unit testing, this is simply known as code coverage or the percentage of the codebase that has unit tests written for it. Further up the test hierarchy, it is the percentage of the actual product that is being tested. This can be quite hard to measure since modern, dynamic applications are not completely deterministic.

What should I be aiming for?

In the ideal world, 100% of your product should be being tested prior to release. As a minimum, you should be aiming to test all the expected user journeys. However, modern software is so complex that there are often thousands of unplanned routes through. And there’s no way to stop your users from taking these. Also, testing takes time, and in a world of CI/CD and rapid releases, it is often impossible to even run all your tests. Let alone test 100% of the product.

How can test automation help?

Test automation can help significantly increase the number of tests you can run prior to each release. Automated tests can be run 24/7, and tests can be triggered as part of your CI/CD toolchain. As a result, it should be a central part of any strategy for increasing QA coverage. However, creating automated tests is time-consuming and hard.

Slow to write scripts: Selenium and similar test automation tools use complex scripts to test your application. Each of these scripts has to be written like a piece of software. It takes time and expertise. Even a skilled Developer in Test can take days to create a good test script. 

Have to adapt scripts for each platform: Once you have a test script it then needs adapting and debugging for each browser and platform it needs to work on. Sometimes, you end up having to rewrite the whole script. 

Need for continual maintenance: Test scripts break easily. Every time you make a change to your UI most of your scripts will break. This is because even minor changes will alter the selectors that the test script uses to interact with your UI. 

The upshot of all this is that few companies manage to achieve even 50% coverage using test automation.

What is Artificial Intelligence

Artificial intelligence is a catch-all term for any computer system that is able to show aspects of intelligence. This includes machine learning, computer vision, and natural language processing.

Machine learning

In machine learning, you are teaching a computer to recognize patterns in data. This can be done in a supervised fashion, where you use labeled data to train the algorithm. But increasingly, attention is being turned to unsupervised and reinforcement learning.

Computer Vision

Computer vision is about teaching a computer to recognize and interpret images. This involves finding each object in the image, categorizing what the object is, and then understanding the semantics of that object.

Natural language processing

Natural language processing is where you teach a computer to understand natural (that is, human) language. This means learning the meanings of words, but also requires an understanding of grammar, idiom, and context.

How can AI tools for software testing help?

Now let us see how these AI approaches can help increase your QA coverage by simplifying test automation.

1. Faster test creation

As we mentioned above, creating test scripts is a painstaking and slow task. Test recorders allow you to speed this up somewhat by recording your interaction with the UI. However, test recorders produce scripts that are even more brittle than hand-written scripts. AI can be used to improve the performance of test recorders. Indeed, intelligent test recorders were one of the first AI tools for software testing. Intelligent test recorders use machine learning and computer vision to make element selection more robust. This means you can create good test scripts much faster.

2. Democratize test creation

As we saw, what really matters is to be able to test the expected user journeys through your application. Traditionally, the product team shared user journeys with the test team, who converted them to test scripts. Here at Functionize, we use natural language processing to simplify this process. Our NLP engine system takes the user journey that the product team wrote and converts it directly into a working test. This dramatically speeds up test creation (up to 11x as fast).

3. Increase productivity

Test maintenance is a bore. More than that, it is a total waste of time and effort. Every hour your test engineers spend maintaining existing tests is an hour less time to increase QA coverage. Indeed, the problem is so bad that many teams are spending up to half their time just making sure their existing tests work. So how can AI tools for software testing help?

Functionize have gone a long way to solving this problem using a combination of AI techniques. These include computer vision, machine learning, and NLP. Our ML Engine builds a complex model of your UI. It learns from the test plans you pass in and also from interacting with the UI during tests. As a result, if things change, ML engine can update the tests automatically. Some changes are subtler and only trigger an error many steps later. In such cases, our Root Cause Analysis system will find the probable cause of the problem, test solutions and present you with the most likely fix. So, you now have even more time available to concentrate on creating new tests and increasing QA coverage

4. Cover more devices/platforms

Modern applications are expected to work on almost any browser and platform. There are many ways to measure the number of browsers/platform combinations. But a conservative estimate is around 32 (more if you need to test different screen sizes). Anything you can do that increases the number of combinations you are testing is going to significantly increase your QA coverage. The way ML Engine models your system means Functionize tests work on any browser “out of the box”. Furthermore, we leverage Google Cloud Compute’s nested virtualization to offer you mobile emulators for all your tests. As a result, simply using Functionize automatically get increases your QA coverage.

5. Simplify test analysis

One key time sink for test engineers is test-analysis. Firstly, they need to decide whether a test failure was a result of a UI change (e.g. was it really a failure). Then they need to work out exactly what failed, so they can notify the relevant development team. AI tools for software testing leverage computer vision, NLP and ML to make this process much simpler. In most UIs, test failures will show up as an unexpected entity on the screen. Here at Functionize, we make it easy to see these. We capture screenshots before, during, and after each step of each test. Our system then compares these with previous tests and highlights any differences. It uses an intelligent algorithm that will ignore style changes or rendering artifacts.

6. Identify missing tests

As we said before, users tend to find all sorts of unusual ways to interact with applications. And these unplanned interactions are often the ones that trigger failures or bugs. AI tools for software testing can record user sessions and identify user journeys automatically. If no test exists for the journey, the system simply creates one and adds it to the test library. This is possible using a combination of machine learning techniques such as K-means clustering and traditional statistical methods like Akaike Information Criteria. You can find out more about this in our blog on automated canary testing.

7. Autonomous test creation

The ultimate way to ensure 100% QA coverage would be a system that uses AI to autonomously test your application. An autonomous (and intelligent) test agent if you like. Effectively, this agent would replace your QA team. The autonomous test agent creates tests automatically based on product design documents and user interactions. When you push new code, the autonomous test agent will use canary testing to check whether this affects the stability. This may seem like a pipe dream, but here at Functionize, we are well on our way to making it a reality!

Conclusions

Artificial intelligence tools for software testing are a key way to increase your QA coverage. They impact this in two ways. Firstly, they make it easier to automate all your existing tests. This means you are able to run all these tests every time you release a new build. Secondly, they allow you to create completely new tests by tracking user journeys.