Cloud testing. Saving precious DevOps time
Automated testing needs major DevOps support. A great benefit of cloud testing is that it can transform your testing, so DevOps can focus on their real job.
How moving testing to the cloud saves scarce DevOps resources
Automated testing has transformed software development. It allows complex UIs to be regression tested in days rather than weeks. However, this comes at a price for your DevOps team who have to support the required infrastructure. We look at how cloud testing can transform your testing and allow your DevOps team to return to their real job.
Automated testing has been about for over 15 years now. Without it, we would probably not have such a rich app ecosystem. However, while automated testing simplifies the regression testing of your UI, this isn’t without costs. One of the biggest issues for any large-scale automated testing set up is the need for constant maintenance of the infrastructure. As a result, your DevOps team will often be pulled off their usual tasks and diverted to helping your test team. Here, we look at the reasons for this, the impact it has, and then show how moving your testing into the cloud solves the problem.
Background
Selenium was the first widely-adopted tool for automated UI testing. Open-sourced in 2004, it quickly became the tool of choice for test automation. While Selenium transformed regression testing, it had some major drawbacks that made it unsuitable for testing large software builds. Firstly, creating tests was extremely slow. Secondly, tests had to be created in Selenese, a domain-specific scripting language. Thirdly, tests could only be run sequentially, with each script potentially taking hours to complete.
Over the next 5 years, Selenium grew as a project. Selenium IDE was released in 2006 which allowed test scripts to be recorded through the UI. Support for additional scripting languages was added with the release of WebDriver in 2007. Then, in 2008, Selenium Grid was released.
Selenium Grid
Selenium Grid was designed to coordinate multiple instances of Selenium running on a number of different servers. Originally, it was a stand-alone product but now it is part of the core Selenium software. One server becomes the hub node which performs all the coordination. The remaining servers (or nodes) run the tests in parallel. Effectively, you distribute your tests across the number of available nodes. This reduces overall completion time linearly with the number of nodes – 10 nodes will complete your tests in about 10% of the time.
There are a few issues with Selenium Grid. Firstly, it is not very scalable compared with many modern applications. It certainly is unable to scale up or down dynamically. Secondly, the configuration is relatively static. Each server is configured in advance with a subset of the required browser instances available. To change the available browsers requires reconfiguring all the servers. Thirdly, although it is able to be run on virtual servers, it isn’t actually designed as a cloud testing application. As a result, it isn’t optimized to take advantage of things like distributed storage, dynamic scaling, and automatic failover.
How test automation impacts on DevOps
Test automation is typically the preserve of a special breed of engineer. These are people who lie in the grey area between manual test engineers on one side and developers on the other. Typically, they are pretty good at crafting test scripts and performing the (sadly obligatory) test maintenance. However, they are not sysadmins or DevOps engineers. This is actually rather problematic when testing modern web applications. There are three distinct issues. Firstly, keeping up to date with all the latest builds of every browser. Secondly, there is a need to create configuration scripts to support things like Selenium Grid. Finally, there is a constant need to maintain the actual infrastructure.
Browser updates
Many modern applications should run on every desktop and mobile browser. Chrome currently dominates the desktop market, but there are 7 other browsers that have >1% of the market (which still means millions of instances world-wide). For mobile, Chrome also dominates, but Safari has 26% of the market and there are 6 other browsers with >1% market share. Each of these browsers tends to release an update at least every 6-12 months. But you can’t rely on users updating every time, so really your testing should also use at least a couple of previous stable releases. This is especially true for mobiles, where operators may limit core software updates. The upshot is, you need to constantly update the set of browsers you are using for testing. This is bread and butter for ysadmins, or you can ask DevOps to automate it as much as possible.
Configuration scripts
As mentioned above, Selenium Grid only allows you to test on the set of browsers available on a given node. The matrix of browser version vs. OS version vs. Platform is immense. And any given node can only really support a handful of browsers. In the ideal world, you would have unlimited hardware budget for your test infrastructure. This would allow you to buy as many servers as needed to cover all the combinations. But the reality is, servers are expensive. So, many testing teams turn to DevOps for support with scripts that will automatically reconfigure servers as needed.
Infrastructure maintenance
The final major issue is maintaining the actual server infrastructure. Most companies nowadays don’t have that much physical infrastructure. They prefer to push as much as possible into the cloud. Often, the test automation team is the exception. An all their test infrastructure needs constant maintenance. That includes software updates, library updates, OS updates, and physical hardware upgrades/replacements. All this takes time and effort and is not really in the skillset of most test engineers.
Cloud testing to the rescue
The obvious solution to the above issues is to move to the cloud. Cloud testing comes in many forms. But in all cases, you save some DevOps hours. We will look at three of the main options and discuss how much time (and money) they can save you.
Roll your own cloud testing setup
As we discussed in a previous blog, creating your own cloud testing setup is feasible with the help of DevOps. The process is the same as with moving anything to the cloud. Choose a suitable provider. This could be Google Cloud, AWS, or Azure, but many smaller providers offer competitive prices and good customer service. Next, decide how many virtual servers you need and how powerful they should be. Then set up your virtual servers. This is usually easier than setting up physical hardware because cloud providers supply pre-configured images for every OS imaginable. Finally, install Selenium and setup Selenium Grid. The main savings you get with this approach are related to the physical hardware. You no longer need to perform maintenance, and software updates can usually be done more easily.
Use a dedicated cloud testing provider
Companies such as Sauce Labs, Lambda Testing, and BrowserStack offer cloud-based Selenium Grid as a service. This means they look after all the setup and maintenance of the test infrastructure. All you have to do is upload your tests and then plan the testing. Using these companies will completely free up your DevOps engineers. But this comes at a price: If you want to run 24 concurrent test sessions, Sauce Labs will charge you $4,799 a month. And, frankly, for many modern applications, 24 concurrent test sessions is simply not enough.
Use an intelligent test agent
Artificial intelligence takes cloud testing to a new level. Here at Functionize, we have spent the past 5 years developing our intelligent test agent. You write your tests in plain English. They need no maintenance. And they automatically work on any browser or platform. All this is possible thanks to combining the processing power of the cloud with some clever AI.
There are two core elements powering our system. NLP engine uses natural language processing to convert test plans written in plain English into fully-functional tests. ML engine uses a combination of machine learning and other AI techniques to model your UI in detail. This allows it to self-heal tests when you update or modify your UI. It also means the tests will automatically work on any browser/platform combination. These elements come together in the Functionize Test Cloud. This allows you to run huge numbers of tests in parallel.
The combination of these things results in a system that makes everyone on your team into test automation experts. More importantly, by leveraging intelligent cloud testing, you make your test team truly autonomous. And don’t just take our word for it. As Vicente Goetten, Executive Director of TOTVS Labs says:
“Before Functionize, our Quality Engineers had to rely on our DevOps and Backend engineers to perform their work, now with Functionize they can not only serve more engineers and cover more parts of the product, but they can also perform their job without the help of other teams.”
Conclusions
Test automation is an essential part of modern software development. However, unless you embrace cloud testing, it comes at a cost in terms of DevOps support. Simply, moving your own test infrastructure to the cloud can save DevOps time. But if you want to free up DevOps completely, you should use an intelligent test agent for all your testing.
