Test infrastructure costs – the true price of Selenium

As every developer and manager knows, the cloud transformed how software was delivered. But over a decade after the beta launch of Amazon’s Elastic Cloud Compute (EC2) service, automated testing often still uses dedicated hardware. Read on to learn how Functionize is bringing the benefits of the cloud to testing, slashing test infrastructure costs and hugely accelerating software delivery.

Introduction

Selenium transformed the world of testing about 15 years ago. This free software finally enabled the dream of many testers ­– truly automated testing of UIs. While Selenium itself is free, to use it effectively costs money. On top of the costs involved in creating working Selenium scripts, you need infrastructure, support software, sysadmin time, and cross-browser test tools. In the following blog, we will look at the true costs of using this free software, explore why cloud-based autonomous testing is better, and explain how the Functionize Test Cloud saves you money.

The true costs of using Selenium for “free”

The traditional approach to test automation relied on free software, such as Selenium. Managers might have been misled by the “free” aspect, but this approach was far from free to implement. So, let’s look at where some of the real costs came from.

• Selenium requires infrastructure to run on. Typically, this means you needed to invest in servers, racks, etc. Furthermore, the hardware is unreliable. So, you are also forced to invest resources in basic sysadmin tasks including keeping your servers up to date, replacing broken components, and solving the myriad problems servers always suffer from. All this adds up to significant test infrastructure costs
• Selenium is not designed to scale. At best, you can use Selenium Grid to set up a small cluster and parallelize your tests. But you will only ever be able to run a few tens of tests at a time. A reasonable test suite for a small web application might easily end up taking a week to complete.
• Selenium tests are hard to create because they use scripting languages. This also means that creating functioning tests is slow, even when you employ (expensive) Developers in Test. Furthermore, whenever your site changes, most of your Selenium tests will break. So those (expensive) Developers in Test spend up to half their time just doing test maintenance.

Overall then, while the software might be free, there are some eye-watering costs associated with using it!

The costs of running your own test infrastructure

Anyone who has looked into buying a server will know they are not cheap. In large part, this is because servers use extremely high-grade components. After all, servers have to be able to run 24/7, often at full load for much of the time. As a comparison, an 8 core Intel Xeon E5-2687W v2 Ivy Bridge-EP running at 3.4 GHz comes in at about $2,100. By contrast, a higher specification desktop CPU, such as an Intel Core i7 with 8 cores running at 4.9GHz, costs $400.

But the CPU is just the tip of the iceberg. Servers need server-grade RAM which, you guessed it, is expensive! Really, you should be looking to store as much test data as possible. Storage is relatively cheap, but you’re going to need a lot of it, probably installed as a RAID array for reliability. Even mundane things like the case are generally way more expensive than your home computer. And if you are being at all serious about things, you will probably need to invest in a proper rack and the associated cooling, power distribution, and KVM. All in, you will be looking at upwards of $50k just on hardware (and that isn’t even going overboard on specifications). So imagine what a complete set of test infrastructure costs.

Software costs

Selenium itself may be free. But what about all the other software and processes you need to develop to create a robust automated testing platform? Developing and maintaining these systems is a significant extra hidden cost. This is over and above the costs involved in just creating test scripts. If you are going to scale at all, you need to be able to manage your tests, have processes for logging and reporting test passes and fails, and a database to store past test results. One of the most valuable things for a test engineer is being able to answer the question: “Is this the first time this test failed like this?”

To create a robust test setup, you need to employ developers to support your test team. You also get into the catch 22 of having to do QA testing on your testing software. Alternatively, you can pay for off-the-shelf software that achieves what is needed. Either way, you are having to pay to be able to use your free software. Then comes the management costs of running all this software stack. You already needed to manage sysadmins for the hardware and developers in test to create the scripts. But you also need to manage the whole process of sharing developer resources.

Cross-browser costs

Any modern web application needs to function equally well in any browser and on any device. Typically, you need to be able to run on round 10-15 different combinations. And if you are being exhaustive, there are many more. Cross browser testing like this has two big hidden costs. Firstly, all Selenium scripts need to be modified for the specific browser/device combination. Some may even need to be rewritten. Secondly, the costs of setting up and managing all the different combinations are high. Especially since many browsers release several major updates per year.

Moving to the cloud

One obvious way to save some money when using Selenium is to move all your testing infrastructure into the cloud. This brings some key benefits.

• Saving on test infrastructure costs. Virtual servers (VS) are much cheaper than real hardware. You can play around with some numbers on the Google Cloud Compute calculator. But roughly you might expect to pay $75 per month, per-VS. Even if you amortized your £50k dedicated server over 5 years, that would be costing you over 10x as much.
• Reduced sysadmin load. Running virtual servers in the cloud still needs sysadmins. But they now have to do much less since most of the hard work is done automatically by Amazon, Azure or Google.
• Accessible from anywhere. One key benefit of running in the cloud is it becomes easy to access your test cluster from anywhere you want. No more need to set up complicated VPNs, etc.
• Easy storage. Cloud-based storage is pretty cheap and has the benefit of being accessible from anywhere. It is also pretty reliable, especially if you select a plan with backup options.

However, there are still issues relating to cross-platform testing, since most cloud operators don’t support that. This is one reason so many people find themselves turning to 3rd parties like SauceLabs and BrowserStack. SauceLabs offer plans with 2 concurrent test sessions for $399 a month, rising to an eye-watering $4,799 for 24 concurrent sessions. While with BrowserStack unlimited plans start at $249 for a single user and rise to $6,229 for 25 concurrent sessions.

A better approach to cloud-based testing

Here at Functionize, we have embraced the cloud revolution with open arms. But we have added a bit of AI magic to create a system that has revolutionized software testing. Our system is based on an intelligent test agent to power truly autonomous testing. The system builds an incredibly detailed model of your application and UI using a combination of AI techniques. These include machine learning, image recognition, and natural language processing. Tests are written in plain English, making test creation accessible to many more members of your team. Once they have been modeled by our Adaptive Event Analysis™ engine, tests automatically work on any browser or device. When you update your application all your existing tests simply self-heal. Without the power of the cloud, our intelligent test agent couldn’t function.

The Functionize Test Cloud

At the heart of Functionize lies our Test Cloud, which is hosted on Google Cloud Compute (GCP). Without the cloud, we wouldn’t have access to the compute power needed to run Natural Language Processing or our robust ML Engine (the two technologies at the heart of Functionize). But using GCP brings several other key benefits for our customers.

• The infinite scale of the cloud means you can run thousands of tests in parallel, with each test getting its own dedicated virtual server. Because each test is running in an independent server, you are also able to test how your load balancer is performing.
• The unlimited storage available means you can keep the entire history of every test you ever run (including screenshots).
• All our tests automatically work on any browser/platform combination. Coupled with GCP’s ability to do nested virtualization means that our system can perform cross-platform and mobile testing with no additional cost or effort.
• GCP has presences in multiple geographic locations, enabling extremely realistic load and stress testing.

Finally, because you can complete your tests so quickly, the Functionize Test Cloud enables proper CI/CD. You can run a complete test suite in just a couple of hours. And of course, you aren't having to pay any test infrastructure costs!

With Functionize, deploy 100 new tests in 6 days vs 74 days with Selenium

Conclusions

There can be no doubt that the future of automated testing is in the cloud. Not only will this save you test infrastructure costs, but it will also help you to create a more robust test setup. However, we go further and say the future of testing is the intelligent application of AI. This can only be done when you have access to the true power of the cloud.