What is smart object selection?

Object selection is the key to UI test automation. In traditional script-based tools, such as Selenium, object selection is typically pretty obtuse; stubbornly tying you to specific variables. In this blog, we explore why this is a problem. We go on to show how smart object selection increases effective test coverage by an order of magnitude.

Test automation is great. Simply point your test suite at your UI and it will find all the errors and bugs, right? If only it were that easy! Selenium can only find the errors that you have told it to look for and as a result, tests often only check a fraction of your UI. There are two issues here. Firstly, writing an exhaustive test scripts that find and check every object on a page is, well, exhausting. Secondly, many objects on a modern responsive and dynamic webpage are simply invisible to Selenium. In this blog, we look at these issues in a bit more detail and then show how smart object selection solves them.

Test automation 101

The importance of object selection

To understand how test automation works, let’s take a look at what it does in simple terms. Essentially, a test automation tool is trying to replicate the actions of a human interacting with the UI. To do this, it will locate certain elements on the UI and perform certain actions.. It will then check that the actual result matches what was expected. This might be verifying the element exists, it could be entering text, or clicking on a button.

Traditional test automation tools like Selenium rely on selectors to identify specific elements in the UI. There are a number of different options to choose from, including CSS selectors, element IDs and even XPaths. Picking the best one for an element is an essential skill for test automation engineers as even minor changes like CSS restyling can alter a large number of these selectors. Also, on many pages there may be a number of almost identical elements. If you rely on selecting the first one that is found you can hit big problems.

Test maintenance

Even the best test automation engineers can get element selection wrong. This is the main trigger for the ongoing need for test maintenance. This euphemistic phrase basically means you end up either updating or flat out rewriting most of your tests each time something in your UI changes. It rapidly becomes a major time sink for test engineers, absorbing up to half their time.

Hidden elements and nested DOMs

Over recent years, as companies have placed additional focus on UX, dynamic and responsive UIs are becoming more and more prevalent. Effectively, the UI is uniquely created for each individual user using embedded content. For instance, many UI pages embed newsletter signup forms from Hubspot or Mailchimp. Shopping baskets and buy-now buttons are often provided by companies such as Mastercard or Paypal. The UI is then rendered by a responsive script according to the user’s screen size and resolution. Even between two identical screen resolutions, you may see minor differences in rendering.

Both these things cause real headaches for traditional test scripts. In particular, embedded content creates issues withnested domain object models (DOMs). This is the logical container that the HTML page exists in. The problem is, Selenium can only really access elements in the parent DOM. This is because the script is running there. To access elements in child DOMs requires you to construct complex XPath queries, adding significantly to the fragility of your script. So, as you can see, element selection is both critical and hard.

Inadequate UI test coverage

One of the impacts of poor element selection is a reduction in test coverage. Selecting elements is hard, so most test engineers tend to only select the minimum necessary for the test. Even on a simple page it is easy to see that this causes problems. Let’s take a simple example of the Facebook homepage.

Now, what are the important elements to test on this page? Obviously, you should check for the login form at the top right. You should also check the account creation form. Incidentally, you will actually end up with at least two different test flows starting from this page. One for the login flow and another for the sign-up flow. But what other elements should you check? Can you spot what has changed in the screenshot below? 

So, how many of you spotted that the Data Policy in the consent text is no longer clickable? And did you spot that the Login help item has vanished from the footer? How about the typo at the footer, which now says “Fecebook” instead of Facebook? All these things are essential parts of the UI, but almost no test script will ever check in that much detail. And indeed, it’s pretty hard for a human to check that level of detail!

Smart element selection

Nowadays, there is an alternative to the rather clunky element selection used in Selenium scripts. Smart element selection uses AI to find the correct elements on a page. The best way to understand this is to look at how Functionize’s Intelligent Test Agent identifies and selects elements. Our ITA leverages a wide range of AI techniques. Let’s look at these in turn, and see how they help power our smart element selection.

Machine learning

We use several forms of machine learning to assist with smart element selection. The system creates a fingerprint for each element, which allows it to be uniquely identified. It combines a whole range of details about each element including classical selectors (e.g. CSS selectors and element IDs) as well as aspects like element visibility, API calls, and server responses.

Computer vision

Our system also uses computer vision to help identify elements on the page. Given that CSS can be changed, this is a key part of smart object selection as we don’t rely on that alone. It is also used extensively by our template recognition and visual testing systems. See below for more details.

Natural language processing

We use natural language processing to check things like the text in buttons. This allows the system to realize that “Add to basket” and “Add to cart” are the same. This sort of smart object selection makes our system robust against most simple UI changes and updates. Incidentally, our system also uses NLP in order to create tests. This takes test plans written in plain English and uses these to train the AI system used to run the tests.

Autonomous template recognition

Often, elements within a UI change dynamically, but their relation with surrounding elements is the same. Take Google calendar as an example. Here, the UI defaults to viewing the current day, week or month (depending on user preferences). These views change according to both the current time and date. But you need to test things like whether the correct current day has been highlighted. We use autonomous template recognition to learn how different elements within the UI relate to each other. This is especially important for testing responsive web pages, which are rendered differently for different screen sizes.

Putting it all together

The key thing with our smart element selection is that it uses so many indicators to identify each element. This means even if a button changes size, moves on the page and has the text changed, we still know what the button is. As a result, all our tests automatically work cross-browser and cross-platform. They also are virtually maintenance-free. Or as we put it, Functionize tests are able to self-heal.

Visual testing

Testing a complete UI is still hard, even if you use smart element selection. NLP makes it extremely simple to create test plans, but it would still be hugely inefficient to specify that each and every element on the page needs to be tested. Instead, we use visual testing in order to check whether there are any UI elements missing. It uses computer vision to examine screenshots taken before, during and after each test step. These are compared with the equivalent screenshots from earlier, successful test runs. We then use an intelligent “fuzzy” visual comparison algorithm to check if anything unexpected has changed. This system is able to cope with style changes (e.g. different borders/shadows on buttons). It also handles elements that move on the screen (e.g. when the length of a field is dependent on the content).

If it spots that something has changed, the test engineer is notified, and the change is highlighted on the screenshot with a red border around it. The engineer is then able to ignore the issue, update the test or report it as a bug.

Conclusions

Element selection is a critical requirement for automated UI testing. However, the limitations of Selenium and other platforms make it hard to get this right. Smart object element selection provides a much more robust alternative. When combined with visual testing, it ensures you can test your whole UI across any browser or platform. By contrast, most script-based approaches are only testing a small percentage of your UI. To sum it up, smart element selection and visual testing help increase test coverage by an order of magnitude.