What is Selenium Grid | Scaling Test Executions

Ever since websites grew beyond simple pages of information with photos and started offering interactive services such as online shopping, social media, and video, proper UI testing has been essential. However, testing web applications is often far more complicated than testing an equivalent mobile app, since there’s a much broader matrix of test variables. For instance, a current web application would, as a bare minimum, need to be tested across 3 operating systems (Windows 8, 10 and MacOS), 4 Browsers (Internet Explorer, Microsoft Edge, Chrome, and Safari) and at several screen resolutions. And if you are thorough, you could easily end up with 50 or more different configurations.

As a result of this, people have always been dreaming of ways to automate and scale the process of testing web applications. One of the early test automation success stories was Selenium, and it’s still a major player today. We discussed Selenium IDE in a previous blog. Selenium IDE is great for creating simple tests, and, coupled with Selenium WebDriver, it can be a great tool for regression testing.  

In this blog, we will look Selenium Grid, a tool that allows you to run multiple instances of WebDriver in parallel. I’ll remind you of a bit of the history of Selenium, explain about Selenium Grid, explore why it’s such an important part of modern web application testing and then look at some of its limitations.

Background

Most of you will be extremely familiar with Selenium. Jason Huggins (one of the fathers of test automation) created the first version of Selenium in 2004 while he was working for Thoughtworks. Selenium was specifically designed to help script and automate the testing of web applications. It quickly became an open source project and additional tools were added to the framework over time. Since then, Selenium has gone on to become one of the most, if not the most, widely-adopted software testing frameworks globally.

The core elements of Selenium are:

• Selenese, a domain-specific language called which is designed to define test cases
• Selenium IDE, a Firefox plugin that allows you to record test cases and test suites and then replay them (but only in Firefox).
• Selenium WebDriver, which coordinates the replay of tests to the browser and can work across many different browsers.
• Selenium Remote Control, a tool to allow you to create automated UI tests in any language and run them remotely against any HTTP website on pretty much any modern browser.
• Selenium Grid, which allows Selenium test suites to be coordinated and run across multiple servers in parallel.

The development and integration of all these projects is coordinated by Selenium HQ.

Selenium IDE and WebDriver

Selenium IDE provides a really elegant way to capture and record test cases via pointing and clicking within a browser window. This is great because it accurately replicates how a real-world user will be interacting with the application. Clearly, as with all UI testing, a good deal of planning has to be done before you record each test case. Selenium IDE is particularly suitable for rapidly prototyping regression and smoke tests. Selenium IDE can also be used to replay the test(s) you have recorded in Firefox. More often though users will choose to export their tests and suites in one of several formats including C#, Java, Python, and Ruby.

Although it has playback capabilities, Selenium IDE is somewhat limited by the fact it only functions as a Firefox Plugin. This is where Selenium WebDriver comes in. Unlike IDE, WebDriver can replay your recorded tests in almost any browser. Currently, WebDriver is compatible with the following browsers:

• Google Chrome
• Internet Explorer 7, 8, 9, 10, and 11 on appropriate combinations of Vista, Windows 7, Windows 8, and Windows 8.1.
• Firefox: latest ESR, previous ESR, current release, one previous release
• Safari
• Opera
• HtmlUnit
• phantomjs
• Android (with Selendroid or appium)
• iOS (with ios-driver or appium)

Because of this, WebDriver is really well suited for running regression tests. Furthermore, WebDriver can also replay tests that were manually created/edited in Selenese.

However, Selenium WebDriver and IDE suffer from two big issues. Firstly, selectors are very narrowly defined during the record process. This has the effect that even quite minor changes to the code, or a minor change of web framework, can render all your test cases useless. Secondly, both run as a single instance, and at any one moment, the server can only run tests on one combination of OS/browser/screen resolution at a time before having to be reconfigured. A good regression test suite has to be run across the full matrix of different server/OS/screen resolution combinations, and nowadays, that often includes testing the responsive mobile version of the application too. This means large regression test suites can easily take many days to complete using WebDriver.

Selenium Grid

Enter Selenium Grid, one of the youngest members of the Selenium family. Selenium Grid allows you to coordinate multiple instances of Selenium running across a number of different servers. Essentially, it provides you with a way to create a distributed test environment for Selenium. It works by distributing the tests across the available number of servers (nodes). This will speed up the execution time roughly linearly compared with running on a single machine. So, if you have 10 machines it will take roughly 1/10th the time to complete your tests.

This makes Selenium Grid a powerful tool for regression testing, where you are keen to expose your code to as many environments as possible but need to minimize the time taken. This also makes it a powerful tool for smoke testing new builds across a standard set of platforms and environments, helping speed up the cycle of Continuous Integration/Continuous Test. 

Another benefit of Selenium Grid is it can be used to ensure you make better use of your test infrastructure. Rather than manually setting up and reconfiguring servers as singletons, Selenium Grid can be used to automate this process and so ensure your infrastructure is better-utilized at all times. This will also reduce the burden testing otherwise places on your devops/sysadmins.  

When you set up Selenium Grid, one server is appointed as the hub. Server nodes are connected to the hub, and the hub maintains a list of which browser instances are available on which node. When a test (or test suite) is run, it will request the specific browser instances it needs. The hub supplies a list of appropriate nodes, and the test is distributed among them.  

Currently, the Selenium HQ team are in the process of deprecating the original version of Grid (imaginatively called Grid 1) and are only supporting Grid 2 in future. The original version of Grid only supported scripts written for Remote Control but was actually a completely different server and setup. Grid 2 is now bundled as part of the main Selenium Server install file and also supports scripts written for WebDriver.

Limitations of Selenium Grid

While Selenium Grid is a really useful tool, it does suffer from some real limitations. These make it less useful than it might otherwise be. Firstly, it has relatively poor scalability compared with many modern applications. It certainly is unable to scale up or down on demand. Secondly, it is pretty static. Each server is configured in advance with a subset of the required browser instances available. If you want to vary this, you have to reconfigure. Thirdly, although it is able to be run on virtual servers, it isn’t actually designed as a cloud-native application. As a result, it isn’t optimized to take advantage of things like distributed storage, dynamic scaling, and automatic failover.

Conclusions

Selenium transformed the process of UI testing web applications by introducing the ability to automate tests. Over time, Selenium has developed to encompass a whole family of test automation tools. Selenium Grid is the newest of these, and in its current iteration, provides a powerful way to distribute your tests across a number of servers. However, as we have seen, it suffers from drawbacks. Some of these are particular to Grid, but some, like the static nature of descriptors, are inherited from the original Selenium.

So, what could be done to make test automation better? Well, the obvious thing would be to create a cloud-native test automation tool. This would be able to dynamically instantiate or delete test servers as needed. Given how relatively simple a UI test server can be, it would be an obvious candidate for containerization. Test scripts could be coordinated from a central location but accessed from anywhere. Test results would be made available anywhere they were needed.  

In the ideal world, artificial intelligence would be used to improve the process. For instance, using Machine Learning you could develop a system that automatically worked out when it needs to trigger additional tests (for instance, if a particular smoke test failed). AI could also create “fuzzy” selectors that didn’t rely on a perfect match. Sound like a bit of a dream? Well, it’s a dream that Functionize is rapidly making a reality!