Functional testing meets artificial intelligence

How an intelligent test agent lets you automate all your functional testing

Functional testing is a vital part of the whole software testing process. This is when you test that your software actually has the correct functionality. Here, we explore how an AI-powered intelligent test agent can transform functional testing of your web application or site.

Background

Functional testing is one of the most important ways to test your software. Indeed, when you ask many people to describe what they understand about QA they will tell you about functional testing. Functional testing is about you verifying that your software actually does its intended job. Importantly, it is black box – you don’t care how it works, just that it achieves the right result. This is different from unit tests and integration tests, where you care how the software works. It is also different from performance tests, where you look at how the software actually performs under load.

In the following blog, we will look in more detail at what is needed to do good functional testing. Then we will look at how you can automate this. Finally, we will show you how the intelligent application of AI can transform your functional testing and dramatically improve your time to market.

What is functional testing

Software Testing Fundamentals defines functional testing as “a type of software testing whereby the system is tested against the functional requirements/specifications.” The important thing to note here is that you are testing against a known set of requirements and specifications. This may seem obvious, but sometimes people can overlook this, especially in a piece of software that has evolved during an agile development process.

One of the key requirements for good functional testing is that it should be focussed on what the end user needs. This means that tests should be generated directly from the user stories, customer journeys, etc. that your product team creates. This is not the same as generating tests from the product documentation. The development team will have written that documentation, and so it only tells you what the software does, not what it should do.

Six rules of good functional testing

1. Functional testing is just about the functionality of your application, not how it achieves it.
2. When you do Functional testing, you are not interested in the performance and reliability of your application. These are left to performance and stress testing.
3. Functional testing is about what your user sees and experiences. But this is not the same as user acceptance testing. Functional testing can happen during integration, not just at the end (though it usually happens at the system testing stage).
4. Functional testing is testing against the business/design requirements of your application, not against the documentation.
5. Good functional testing should test all the moving parts of your code. This requires careful thought and test planning.
6. Functional testing should use carefully designed test data with known expected outcomes. This should come from the requirements/design.

The importance of functional testing

It is fair to say that functional testing is the most important form of testing for many applications. Users may cut you slack if your application is not performant. But if it fails to do what it is meant to they will never use it. Once you have an application that is functionally correct, then you can focus on improving performance. But before that point, performance is irrelevant. In many ways, this is the philosophy behind test-driven development.

Proper functional testing is about ensuring that your application meets its requirements. It may pass all unit tests and non-functional tests with flying colors. But broken functionality means the whole application is broken. Getting functional testing right requires close collaboration between your product and testing teams. The product team has to define every user journey through the system and the test team then have to check these give the expected outcome.

Automating functional testing

Functional testing can be done manually using traditional test plans. However, it is much better if you can automate it. Test automation tools will help make your life easier, especially if you have a complex application. Nowadays, many applications are browser-based rather than native. Automated testing for web applications is relatively common. However, it suffers from many well-known problems.

Problems with traditional functional testing tools

Modern web applications are often highly complex, producing dynamic UIs with embedded widgets and content. During the development cycle, they will go through many design changes. In most cases they will be deployed using CI/CD, meaning they will remain functionally dynamic throughout their life. A final key issue is that responsive design is now a de facto requirement of any good application. This allows it to display correctly on any size of the screen. However, all these things are a nightmare for Selenium.

Writing test scripts is slow

Test scripts need skilled Test Automation Engineers or Developers in Test to get them right. Even a simple script takes hours to write and debug. Test recorders try to simplify this process, but the scripts they produce are more brittle. They also still require manual debugging and improvement.

Scripts have to work cross-browser and cross-platform

Adapting Selenium scripts to work cross-browser and cross-platform is a painstaking and slow task. Essentially, you have to rewrite and debug the script for combination of platform and browser. As a result, it takes even competent test engineers days to produce a few test scripts that work cross-browser.

Dynamic sites and nested DOMs are a nightmare for Selenium

Domain object models (DOMs) are the logical containers within your web application’s UI. Many modern sites embed dynamic content from 3rd parties. These include things like a PayPal “Buy Now” button, or a newsletter sign-up popup form. All these objects exist inside their own DOM embedded inside your application. Sometimes, these DOMs can be nested many layers deep. The problem is, legacy test scripts can only reliably access and interact with objects inside their own DOM. The only way to accurately access embedded objects is to use complex (and slow) XPath queries.

Test maintenance takes half your time

Last, but by no means least, is the test maintenance problem. Test scripts are notoriously brittle. Each and every time you update your UI, you expect most of your Selenium-based tests to break. This is because every change makes it harder for the script to select the right element. Even a simple CSS change can cause all the selectors to change.

Sometimes, the script simply fails to find the element at all and will fail instantly. Other times, it will actually select the wrong element. This causes real problems since it leads to failures that only show up many steps later. In the worst cases, it could invalidate the test altogether. In both cases, you have to manually fix the test script, which is why test maintenance ends up taking half a test automation engineer’s time.

AI-powered functional testing

So, what can we do to improve functional test automation? Firstly, you need to simplify the process of taking user journeys and converting them into tests. Secondly, you need to be able to test even the most complex of web applications on any and every combination of browser and device. Thirdly, you need your existing tests to work even when you make changes to your UI. So, how can Functionize’s intelligent test agent achieve all these?

Simplifying test creation

Our NLP engine allows you to upload test plans written in plain English into our system. Our intelligent test agent then takes batches of these test plans and uses them to learn how your system works. Having learned, it then converts the plan into a working test. You can use structured test plans (like the formal plans produced by test management software). But alternatively, they can also be unstructured, like classic user stories and user journeys. As a result, it is easy for your product team to help generate working functional tests.

Coping with complex sites

Our ML engine uses a combination of several AI techniques in order to create a detailed model of your UI. Importantly, this model fingerprints elements in the UI using multiple methods. These include classic CSS selectors but also cover things like element visibility, location on the page (and relative to other elements), Xpaths, previous locations, etc. The result is a system that can seamlessly cope with nested DOMs and dynamic content. Even better, when you create tests with ML engine they automatically work cross-browser and cross-platform. To top it all off, all Functionize tests run in our Test Cloud. This allows you to run thousands of tests in parallel, dramatically improving the productivity of your testing team and helping cut time to market.

Slashing test maintenance

AEA™ brings a major benefit to test maintenance. Element fingerprints are much more robust than the selectors used in classical test scripts. So, Functionize tests simply self-heal when you move or restyle elements. When changes lead to more complex test failures, our Root Cause Analysis system is able to track the change that triggered the failure, even if this was many steps back. It can then suggest the most suitable fixes and even test these to choose the best. All you have to do is a single click to accept the suggestion.

Conclusions

Functionize’s intelligent test agent makes it simple to automate all your functional testing. You don’t need to employ an army of Developers in Test to create new tests for you. Because we leverage the power of cloud computing, your functional testing completes far faster. And Functionize tests can self-heal when you make changes. This transforms your functional testing and helps deliver good software quicker.