AI test automation: 5 ways to avoid budget overflow

Software testing is an expensive business. In complex projects, it can often cost up to 40% of the total budget. Here we look at 5 ways to avoid budget overflow through AI test automation.

Testing is essential if any software product is going to succeed. End-users dislike buggy and unstable software with a passion. Ship a broken product and your users will vote with their feet. The trouble is, the cost of proper testing grows as applications become more complex. For modern web and mobile applications, testing can cost 40% of the overall project budget. Get it wrong and you can quickly see budget overflow (and a whole heap of pain). In this blog, we explore how AI test automation can help control the costs of testing and thus prevent cost overrun.

Background – why is testing so expensive?

Way back in the early days of computing, running a program meant booking time on an extremely expensive mainframe computer. As a result, programmers were under intense pressure to ensure their code worked perfectly. But as computers became cheaper and more pervasive, programmers started to concentrate more on features than whether code worked. However, working code was still a requisite if you were aiming to make money. And so, we come to software testing.

Traditional software testing means manual testing. This is extremely labor-intensive and inefficient. It is ideal if you need to do exploratory testing (the sort where you poke it until it breaks). But most testing is about verifying that the happy path is working (smoke tests, regression tests, acceptance tests, etc.). Clearly, this sort of testing is quite boring, long-winded, and repetitive. In other words, it is perfectly suited to automation.

Test automation is a flawed creature

Test automation is primarily about testing an application’s UI. The earliest widely-adopted test automation tool was Selenium. In Selenium, you use a scripting language to select and interact with elements of the UI. You then check the result to see if it is what you expected. There are three fundamental flaws with this approach:

1)    You need expert developers to create robust test scripts. 

2)    Writing test scripts is very slow. 

3)    Test scripts are inherently brittle and prone to failure.

Expert developers

Test scripts are complex creatures. You need to be certain you are selecting the correct element every time, which is especially challenging with modern dynamic UIs. But writing test scripts requires different skills to writing good code. There are many good developers out there, but very few of them have the kind of mind that makes a good test engineer. As a result, “developer in test” has become one of the most high-paid jobs in the testing industry. According to Glass Door, the AVERAGE salary for a Software Engineer in Test is $115k. By contrast, a Senior Test Engineer only receives $105k.

Slow to create scripts

Writing test scripts is slow, painstaking, and hard. At every stage, you have to debug and test your test script. Then you have to modify the script for each browser and platform you need to test it on. As a result, even a simple test script takes a skilled engineer several days to get right.

Test scripts fail

Test scripts are astonishingly brittle. Every time you make a change to your UI they tend to break. This is because even minor changes will affect the selectors that the test script interacts with. The upshot is that most test engineers now spend half their time performing routine test maintenance.

What is AI test automation?

AI test automation involves using artificial intelligence to enhance and improve test automation.

Intelligent test recorders

One of the big problems with test automation is creating tests. AI-driven intelligent test recorders try to make this easier. Test recorders are nothing new ­– they’ve been about since Selenium IDE was launched over a decade ago. A test recorder simply records a user interacting with a UI and converts this into a script. This script then needs to be modified and debugged by hand.

Intelligent test recorders take this a stage further. They use techniques like machine vision to allow “fuzzy” selection of elements. Some also use machine learning to learn what each element you interact with actually does. The result is that they produce much more robust scripts that need far less debugging. The upshot is you are able to create scripts much faster.

Intelligent test agents

An intelligent test agent takes over the task of creating, running, and analyzing tests. To do this, it uses a combination of several AI techniques. These include machine learning, computer vision, natural language processing, and template recognition. Here at Functionize, our intelligent test agent allows you to create your tests in plain English. Our NLP engine will take your test plan and generate your tests for you. Our ML Engine system uses these tests to learn how your application works. As a result, the tests are almost maintenance-free. They also work across any platform and any browser. We also make test analysis a piece of cake, using fuzzy visual matching to highlight unexpected results in tests.

Five ways AI test automation reduces budget overflow

As we have seen, test automation is expensive. Arguably, the single biggest expense in your software development budget. So, how can we use AI test automation to reduce and control that expense?

1)    Speed up test creation and increase coverage

Traditional test scripting is slow. AI test automation is much faster. Intelligent test recorders can allow you to create tests in just a few hours. Functionize’s NLP engine takes this even further, allowing you to create and model round 50 tests in a single day. The more tests you automate, the more time you free up for exploratory testing and the fewer resources you waste. Your aim should be 100% test automation for your routine and regression testing. This will allow you to save money by reducing resources. Both intelligent test recorders and intelligent test agents help you to achieve this.

2)    Reduce test maintenance

Intelligent test recorders can reduce test maintenance because they are robust to small changes in the UI. Functionize’s self-healing tests go a stage further and reduce test maintenance dramatically. This is because AEA™ learns your system in detail. This makes it robust to almost all changes and updates to the UI including moving elements, renaming buttons, and updating styles. Where there is ambiguity, our Root Cause Analysis engine will identify the root problem, test potential solutions, and present the most likely to you for approval.

3)    Cross-platform as well as cross-browser

Modern applications need to work cross-platform as well as cross-browser. Almost all applications have both mobile and web versions. Creating tests for all these is time-consuming and expensive. Mobile testing in particular often requires you to pay external companies. Functionize’s tests run in virtual servers on the Google Cloud Compute platform. GCP allows nested virtualization, which in turn enables mobile emulation. As a result, Functionize's solution removes the need for third-party cross-browser tools.  

4)    Less test infrastructure

Selenium and related tools require their own dedicated infrastructure to run. This is a sunk cost for you and your team. You have to pay for the hardware, keep the OS and software up to date, and pay for DevOps and Sysadmin support. Or you can use a cloud-based platform such as Sauce Labs. But that doesn't come cheap.

By contrast, all Functionize tests are run in the cloud. This gives you the reassurance that all browsers and platforms are running the latest builds. Furthermore, we don’t impose artificial limits on you. If you need to run hundreds of tests in parallel, our system will let you. Indeed, because the system is powered by machine learning (among other things), the more tests you run, the better it learns your application.

5)    Proper load testing

Load testing is a must to prevent and control cost-overrun when your application goes live. There are two forms of load testing. In the first, you are looking to understand what resources your application consumes under normal load. In the second, you are trying to stress your application to learn how it fails. The traditional approach to this is to use tools that generate multiple HTTP sessions, or multiple parallel API calls. These are all very well, but they are pretty artificial. By contrast, Functionize allows you to launch realistic user sessions from multiple geographic locations. Each user runs in their own VS, so your application load balancers will be doing their job. The result is, even prior to launch, you can have a pretty accurate idea of how your system will behave.

Conclusions

AI test automation can help you control budget overflow. If you use Functionize’s intelligent test agent, you can go a stage further and slash your test costs and dramatically increase your test effectiveness. But even intelligent test recorders are better than relying on legacy test scripting.