Local vs Cloud Execution

Automated web tests can be run in different environments. This often leads to a debate — Should tests be run on local browsers or run in the virtualized cloud?

Here’s why tests should be run in the cloud:

1. Only the cloud is powerful enough to process deep learning based tests

Functionize uses Deep Learning to keep tests resilient and continuously improve them over time. Deep learning means that our machine learning models are trained on such large amounts of data that our technology has independent decision-making. This means that tests run with much higher accuracy using rich contextual data collected from your application under test. This information ranges in the tens of millions of data points per single test execution and includes such things as the visual state, positional, relational, layering, readiness of elements, scrolling and timing data. 

This enormous amount of data is what allows Functionize to identify elements and run tests with near humanlike intelligence, to make decisions at runtime and adapt to the changing nature of your application under test. In order to process such large data, you need the compute power of the cloud. Traditional automated tests pale in comparison as they identify elements using a single hardcoded attribute, like the CSS or Xpath selector. Without the power of the cloud, it’s impossible to run tests with deep learning. And without this deep learning, your tests fail and break frequently.

2. Local execution slows down the delivery process

Tests powered by deep learning have a huge advantage in slashing time wasted in test maintenance. Testing in a local environment greatly limits the amount of data that can be processed, which leads to broken tests. It’s extremely frustrating to see your automated test fail even though the exact same test passes when run manually. This brittle nature of automated tests is a complex problem, and it requires the power of deep learning to solve it. 

Running tests locally may be appealing to those who wish to run in lower environments, but what’s the point of running tests earlier if those tests fail to run successfully? Local execution may promise faster feedback earlier, but that feedback is unreliable if tests don’t work. Eventually without testing in the cloud, your QA and Dev teams will get consumed by the test debt problem, and bring your delivery process to a screeching halt.

3. Local execution requires clunky setup and upkeep of testing infrastructure

Running tests locally requires you to set up your testing infrastructure ahead of time and keep it up to date with any upgrades in the future. When you run with the Functionize Test Cloud, there’s no need to manage infrastructure. No need to call your DevOps team to help spin up machines to run tests. There’s no configuration necessary, so you don’t have to worry about how much RAM or disk space is necessary. You can be confident that your tests will run in a clean CI/CD environment without being encumbered by any environment or configuration related issues. 

4. Running tests in the cloud is more cost-effective

Running tests locally does not mean that executions are free. Even if a test automation vendor does not charge for local executions, your organization still has to pay for the cost of running tests on a VM. This means that the cost per local execution still shows up in the invoice from AWS, Azure, or Google Cloud Platform. If you compare those prices, it’s much more cost-effective on a per VM basis to run tests in the Functionize Test Cloud. Luckily, we use machine learning to search and find the lowest price VMs at runtime. Functionize also takes care of provisioning and deprovisioning VMs at runtime, so we’re not racking up VM costs for machines that are not in use and we can pass on those cost savings to our customers.

5. The cloud allows you to scale executions and run them even faster.

By running tests in the Functionize Test Cloud, you can spin up multiple VMs at once to your tests concurrently. If you choose to run 10 tests locally, it will take 10x longer because you can’t run more than one test at the same time. With the Functionize Test Cloud, you can run an unlimited number of parallel tests. This drastically reduces the amount of time you would need to wait for execution results to be ready. This truly allows you to get testing feedback faster, especially since your tests are less likely to break and can run in parallel.

If the developer wishes to run tests in a local environment before merging code. Some testers may be used to running tests locally to test earlier in the process. But, this slows down the delivery process. Why? Because tests that don’t use deep learning break more often. So when tests break in a development environment, you slow down developers who waste time debugging their tests. This is why unit tests are more valuable in lower environments, so that you can do faster checks when developers check in code.

Are you still not convinced? If you can’t live without local runs…

The Functionize Command Line Interface (CLI) allows you to connect with non-public sites so you can tunnel to a local development environment instead of setting up a whitelist for our servers. The CLI offers most of the functionality of our API endpoints plus a few extra commands from a simple interface. So, if you want your developer to run tests, they can use the CLI to tunnel to their local development environments. However, we believe that developers should spend less time reviewing test results, and more time actually coding. Running in the cloud allows testers to run and review tests, and waste less time in test debt in the process.