Top 7 Reasons to Move Your Tests to the Cloud | What You Are Missing if You Don't

Cloud computing is a truly transformative technology, ranking alongside such developments such as steam power, the internal combustion engine, and electricity. The cloud powers almost everything in our modern lives, from social media and video streaming to mobile apps and Internet search. Without the cloud, companies such as Google, Amazon, Facebook, and Netflix simply couldn’t exist.

Moving operations to the cloud has been a disruptor for many areas of business, and software development is no exception. From code repos such as GitHub or Bitbucket to cloud-based IDEs such as codeanywhere or Amazon’s Cloud9, the cloud is increasingly being used to accelerate the software development cycle.  

Software testing, in particular, can benefit from running in the cloud. In this blog, we’re going to look at the seven key benefits of running your tests in the cloud. If you’re not already using the cloud for your testing, hopefully, this will encourage you to explore the options. If you are already using the cloud, maybe this will give you some pointers to things you haven’t come across yet.

1. Near infinite scalability

Scale is probably the single most obvious benefit of moving to the cloud. Big cloud vendors such as Google Cloud and Amazon AWS will provide you with almost limitless resources on-demand and you can scale up (and down) as you need. This dynamic scalability is particularly useful if you do periodic releases as it allows you to scale up your test resources during the pre-release test phase. The real benefit of this is being able to run massively parallel tests. For web application testing, in particular, this allows you to test across the full range of OS/browser combinations. This also allows you to use your testing resources more cleverly. If you have a test failure you can keep running your other tests while you explore the failure in more detail.

2. Speed

In the early days of cloud computing, the speed of your virtual resources was patchy at best. To maximize revenues, cloud service providers would massively overcommit virtual servers onto physical hypervisors. Nowadays things are very different. While you can still find ultra-cheap virtual servers for a few dollars a year, the big providers like Google Cloud and Microsoft Azure now offer blisteringly fast hardware with guarantees about availability and overcommit ratios.  

When you move your tests to the cloud you benefit from the scalability discussed above. You are also able to utilize much faster hardware than would otherwise be the case. Taken together, this means your test matrix can run faster by using more resources in parallel and hence will complete much sooner. In turn, you will then find any bugs much quicker and thus can reduce the time to market for your software. As you well know, the faster you turn around your tests, the more streamlined your development cycle becomes!

3. Cost

In any company, the bottom line is what matters and running your own test infrastructure comes expensive! Not only do you have to buy the server hardware and pay for the required power/cooling, you also need to employ in-house sysadmins to look after it all. As a result, management can end up viewing test infrastructure as something of a sunk cost. The usual compromise is to try and save money by reusing old build and production servers for running tests.

However, if you use cloud-based infrastructure, you only pay for the resources you actually need and use. If your release cycle allows for it, you may even be able to benefit from things like spot pricing, meaning you get a significant discount on the list price. Furthermore, the scalability and parallelism discussed above mean that you will be able to free up the resources quickly and stop paying for them.

4. Heterogeneity of test environments

Almost all software needs to be able to run in a variety of different environments (different hardware architectures, different OS stacks, etc.). This is especially true for UI testing of web applications. Here, in addition to needing to test on a range of hardware and OSes, you also need to be sure the application runs well at different screen-sizes and across the whole range of current browsers. You even need to test the impact of different versions of libraries and frameworks that might exist.

Frankly, few companies can afford the cost or manpower needed to set up and run test clusters that cover the entire matrix. Even using Xen or KVM to run different OSes can only take you so far. Enter the cloud. With a cloud-based test setup, you can run on a variety of hardware, as well as being able to spin up any OS and software stack you need. Better yet, you can run all these tests in parallel. You can also set things up so that as new versions of frameworks or Browsers are released, these are automatically added to the test grid. This is particularly useful when you are dealing with a browser like Chrome that pushes significant changes every few weeks.

5. Playback of test results

One of the most powerful features offered by cloud-based test solutions such as the Functionize Test Cloud, is the ability to model and playback your test results visually. For complex test scripts, this is an essential feature that can greatly simplify the process of analyzing any test failures. It also has the incidental benefit of giving you useful videos of your application which you can use in marketing or investor relations! Obviously, it is possible to set up your own infrastructure to do this, but you will end up needing to provide additional resources in terms of storage, compute power, etc. You will also need to create a system to index the recordings and associate them with the relevant tests. Again, this will have a resource cost in terms of developer time.

6. Artificial Intelligence/Machine Learning

One thing that cloud testing offers that’s almost impossible to implement on local hardware is the ability to use machine learning (ML) to create a smart test suite, capable of running self-healing tests and able to learn from previous test scripts in order to automatically generate new test suites.

Machine Learning is a form of Artificial Intelligence (AI) that uses neural networks to emulate the action of the human brain. Each neural network (or model) is trained to recognize certain features depending on a set of inputs. The reason ML is so hard to implement on local hardware is the scale of compute resources needed to train the model and to keep the model up to date.  

Here at Functionize, we use ML for intelligent data modeling. Rather than providing absolute selectors in UI test scripts, our system uses fuzzy selectors. This, in turn, makes the test scripts robust against minor changes that would break a typical Selenium test script. We also use AI to power our visual testing tool. Rather than trying to do a pixel-by-pixel visual match (which can easily break down when doing cross-browser testing), our approach uses computer vision to learn what the page looks like as a human would view it. This then allows us to automatically parse content changes and to update test scripts accordingly.

7. Test Debugging

The final benefit of running your tests in the cloud is the ability to use the additional resources and data to debug your tests. For instance, in a UI test, it’s often important to verify that a page has correctly loaded before proceeding to the next step. If you fail to provide this check your test may start throwing failures that aren’t valid. Because you are able to visualize all your previous test results and can use visual playback to check what was happening, you can identify this sort of problem much more easily.

Some tests may also be unreliable – sometimes passing, sometimes failing. I have worked for companies before where the test team have told me “Don’t worry if that test fails, it just does that sometimes. Try running it again.” Rather than accept this sort of absurd situation, cloud-based testing allows you to debug these dodgy tests properly. Partly this is just a function of the ability to store and process data and metadata from every test run you do. But it’s also helped by the limitless compute resources you have available.

Conclusions

The cloud powers almost everything in our modern life. After cloud emerged in the middle of the last decade, developers were quick to exploit its potential, and it has been a huge driver for all the myriad apps and web services that now enrich our lives. And yet software developers have been slow to adopt the cloud to power the development process itself. This blog explored some of the reasons why you should consider using the cloud for running your tests. Hopefully, it will inspire you to explore other ways in which you can leverage the power of the cloud to speed up and improve your development cycle.