Automated functional testing 101

Automated functional testing involves verifying that your application actually does what it is designed for. It is one of the most essential forms of testing, but getting the automation right can be hard. Read on to learn more and to discover how Functionize can help you.

What is Functional testing?

Functional testing involves verifying that your application does what it’s designed to do. More specifically, you are aiming to test each functional element of your software to verify that the output is correct. This is black-box testing because you only care what the system does, not how it does it. In order to do this, you have to view the software from the user’s perspective. In other words, you must interact via the user interface.

It is important to understand a few things that distinguish functional testing from other forms of testing.

1. Unlike some testing, functional testing isn’t specifically about finding bugs. What matters is checking that the software meets the specification.
2. Functional testing is not user acceptance testing. UAT only happens at the very end of the development cycle, whereas functional testing should start as soon as you have a working UI.
3. Functional testing is never ad hoc. You must plan carefully and check the plan against all the requirements. Typically, this means cooperating closely with the product team.
4. Functional testing should use data that has been designed to give sensible (and suitable) results. You are not trying to trigger edge conditions.
5. You don’t care about the usability, performance, or stability of the software unless those affect functionality.

Why is functional testing so vital?

Most software testing is about reliability, usability, and performance. However, functional testing is when you verify that the software actually works. That is, that it actually does what you need it to. It is one of the ways of trying to avoid a return to the “software crisis” of the 1970s and 80s. This was characterized by the software being delivered late, with poor quality, and missing functionality. Without functional testing, you can’t verify that your application meets its requirements. It may pass every unit test and non-functional test perfectly. But if any of the central functionality is broken, the whole application is broken.

What are the challenges for functional testing?

We have all experienced apps where some key function fails to work as we expect. Even giants like Google and Microsoft are not immune to the problem. This shows that functional testing is not completely straightforward. Let’s look at the key challenges.

Planning. You have to plan your functional testing really carefully. Each user journey should be tested thoroughly. You need to include ones that might be unusual or non-obvious such as password resets, users making account updates, etc. This requires close cooperation between your product team (who understand what the application is designed to do) and your test team.  

Execution. Functional testing requires you to perform a large number of detailed tests. Usually, these tests need to go end-to-end. That is, they must complete a specific user flow in its entirety. This makes functional testing slow to perform manually.  

Analysis. At each stage in the test, you need to check that the application is doing what you expect. This means checking the UI displays the correct information and that the final result is correct. You sometimes need to verify that actions on the frontend have triggered the correct result on the backend. For instance, in a shopping application, you need to check that new orders reach the backend system.

What is automated functional testing?

You can do Functional testing manually of course. But it is much better if you automate it. Automated functional testing involves getting a computer to run your functional testing for you. Automating functional tests brings some key benefits.

Firstly, you can run tests 24/7 without human intervention. Automated tests also run faster than manual ones. Taken together, this means you can complete your testing in a much shorter time Secondly, computers follow the plan exactly. They don’t make errors such as skipping test steps or using the wrong data. This is a definite problem with manual tests. Thirdly, you can tweak tests simply by changing the test data. This allows you to easily run sequential tests with slightly different data in order to compare the results.

How do you create automated tests?

The traditional approach for test automation is to write a Selenium test script. Selenium is the Granddaddy of test automation tools. It has been around for over 15 years now. A Selenium test script contains a series of test steps. For each step, you select elements in the UI and interact with them (e.g. pushing a button, filling in a text box, selecting from a dropdown). You then check the result and compare it with what should have happened. When you run the script, Selenium Web Driver converts this into interactions with a real browser.

What are the issues with using Selenium?

Selenium was very impressive when it was first released. It was the first system able to be used for testing any browser-based UI. However, it is far from perfect. So let’s look at the 3 main issues with it.

Slow test creation

Test scripts are pieces of software. This means you need to develop them like any other software. Typically, you develop the script incrementally. At every stage, you test and debug what you have coded. The big challenge with Selenium is to reliably find the correct element on the page. You do this using element selectors. These selectors must be unique. This ensures that only the correct element is selected. You need to use immutable selectors, so each time you run the test, the same element is chosen. Often, a test engineer will have to construct quite complex selectors. And some dynamic sites make it remarkably hard to get this right.

Test maintenance

Each time the UI changes, there is a high chance that some of the element selectors will also change. The result is, your test will now fail to complete. The only solution is to manually edit all the scripts that now fail, update the selectors and then test and debug the script. Typically, the selector just doesn’t exist and so the test fails. But sometimes a selector changes such that the wrong element is chosen. Now, the failure can happen many steps later. These cases are trickier to track down and fix. The upshot is, test maintenance can take 40-50% of your time.

Poor scalability

With modern applications, you always need to test both on desktop and mobile devices as well as on multiple browsers. This means you need to repeat every test several times. Selenium predates cloud computing, and so was never optimized to run in the cloud. Selenium Grid allows you to coordinate multiple instances, but it is still relatively hard to scale. There are companies that allow you to run Selenium tests in the cloud, but they often charge a lot for this.

How can AI improve things?

At Functionize, we leverage multiple forms of AI in order to address the issues with Selenium tests. We describe our approach as an intelligent test agent that helps you create effective, maintenance-free tests that can run in the cloud.

Test creation. Our NLP engine takes the pain out of creating tests. You pass it a set of test plans written in English, and it will generate a fully-functional set of tests. NLP engine can even create tests from plans written unstructured English, such as user journeys. The result is you can create new tests in a fraction of the time it takes with Selenium. 

Test maintenance. ML Engine is the brains behind our approach. It uses a combination of machine learning, computer vision, and natural language processing. ML Engine assigns every element in your application a unique fingerprint. These are far more robust than the element selectors in Selenium. As a result, changes like restyling, UI layout, and element renaming don’t break our tests.  

Testing at scale. We built Functionize using a cloud-native serverless architecture. As a result, the Functionize Test Cloud allows you to run large numbers of tests in parallel. Our advanced orchestrations even allow you to avoid issues with tests “colliding” with each other. Another key advantage is we offer mobile browser testing as standard by using Google Cloud’s nested virtualization capabilities.

Summary

Functional testing is a central component of your QA strategy. Functional testing verifies that your application is actually working. Where possible, you should automate your functional testing. But as we saw, there are some issues when you use Selenium for automated functional testing. The solution is to use Functionize’s intelligent test agent.