How you should be Load Testing | Why Real User Sessions are Necessary

Load testing remains as important as ever, both to ensure that you deliver consistent performance to users and to allow you to understand how your system needs to scale to cope under load. One of the big advances Functionize brings is improved load testing. Rather than rely on multiple, relatively static HTML sessions, Functionize executes fully-formed user interactions at a realistic pace, which can be configured to correspond with action sequences from users in the field. Load testing is done by executing complex, actual-workflow functional tests against your website. High-volume simulations are run from dispersed geographic locations. This makes our solution altogether superior to tools that merely run excessive page-load requests and form submissions.

Load testing is the process of assessing how well your app or site performs when subjected to load. This is essential because it will show you the point at which your app will no longer remain stable. And as we all know, if an app keeps crashing or freezing it will start to hemorrhage users. Under load, some apps will suffer a steady and predictable decline in performance, with server responses becoming increasingly slow until (eventually) you see some form of timeout. Other systems may handle increased load in a relatively stable fashion until they suddenly fail. 

The aim of load testing is to understand two things. Firstly, how do overload failures manifest themselves in your system and secondly, how much load can your system handle before the performance deteriorates enough to cause issues for users. Where your system suffers a gradual decline in performance, this second factor usually comes down to understanding how slow the system can run before customers get fed up and walk away.  

For modern cloud-based apps, load testing can be much harder. In many cases, your system will simply scale up to cope with unforeseen load. However, scaling up comes at a price, both in terms of resources and in bandwidth needed. Both of these will affect your bottom line, so it’s always worth knowing what to expect. Also, there are often system elements that can’t scale so well, such as databases or custom libraries. The very fact that the app resources can scale on demand may suddenly expose these parts of your system to unsustainable loads. So, proper load testing of cloud apps is also important both to understand how to scale your system initially and the likely costs you will incur under increased load.

How existing load testing tools work

Historically, load testing was done semi-manually, with test engineers using scripts to create as many HTML sessions as possible, with wget to fetch content and making API calls to send data to the system. Their aim was to try to place a known load on the backend servers. A typical test might see the engineer use Python to make repeated calls to a simple shell script that sends API requests and fetches page objects from the server.

Most existing automated load testing tools build on this approach. They try to place the system under some sort of stress by repeatedly loading pages, submitting forms and making other HTML requests. However, the problem is these tests are often relatively static. Usually, they run from one location (or maybe a handful of locations), they use a list of predetermined page requests or form interactions and, at best, use some sort of pseudo-random distribution to generate the intervals between requests. This means they are not placing a realistic load on the system under test.

The limits of faking users

On paper, a test system that generates HTTP requests, submits form data, and loads the API might seem to give a reasonable way to test your system under load. However, there are some significant limitations, especially when it comes to testing cloud applications. One of these is that sessions will all be coming from one location. This means that your load balancer may not be accurately distributing the load among your servers or containers. The second issue is that new sessions may arrive unrealistically fast and because no account is taken of how a human interacts with the system, forms are completed and submitted instantly. These factors mean that your system may appear to fail sooner than it would with real users. The third issue is the most significant though. Despite the best efforts of product managers everywhere, real users are unpredictable. Even a relatively simple web application will have thousands of possible routes that a user may take to navigate through it. Each of these routes will place subtly different demands on your backend system.

You can ameliorate the first two problems relatively simply (though not necessarily completely solve them). For instance, you can make sure your load tests are executed from several discrete locations. That will help ensure they are correctly handled by your load balancer. You can also add some random delays to events in your tests in an attempt to simulate how a real user might fill in a form or similar. But experts in network simulation will tell you that accurately modeling user behavior with statistical distributions is a (very) hard problem. However, even if you do both these things, you cannot address the human element in how users interact with the system. For that, you need real user sessions.

How Functionize executes load tests

Functionize takes a more intelligent approach to load testing. Our load test solution takes your existing Functionize tests and runs these in parallel from hundreds or thousands of VMs hosted in Google Cloud locations globally. Functionize tests are designed to interact with your site in the same way a real user would. Therefore, they accurately simulate real users accessing the site. Our tests take into account things like the time it takes a user to complete a form, the time taken for images to load and the unpredictable delays in server responses. In other words, all the things that affect how real users interact with the system.

When you launch a load test from the Functionize dashboard, you specify the test suite to run, how many simultaneous sessions you want to run and how long the load test should last. The system then launches a sufficient number of VMs to accommodate simultaneous sessions and repeatedly runs the chosen test suite across these VMs. Once all tests in the suite have completed, the system restarts the suite execution again till the desired load test duration has elapsed. 

Using this approach provides the most accurate possible load test process for your system. In effect, it’s exactly like that number of real users simultaneously accessing your system. Furthermore, because of the tests loop, you can be sure that the load will be sustained for the duration of the test. This gives you real confidence that your system can cope under that load, as well as allowing you to measure the impact in terms of scale, cost and performance of differing load levels.

Specifying the tests

Creating a load test is extremely simple using the Functionize interface. As already mentioned, you need to specify which tests you want to run. The load test module can run entire test suites (aka projects), specific modules or tests that match a tag. You may choose one of your existing Functionize tests or projects, or you may decide to create a new test specifically for use with load testing. This second approach may be particularly useful when you wish to stress a particular element of the system, for instance, the login or account creation flows. Or you might go through your tests tagging selected ones for use in load testing.

You then specify which environment you want to test. In general, for load testing, this will be the live environment. You also select the region to run the tests from. Because the tests are run on Google Cloud, they can be run from any geographic region to suit the profile of your users. You could even create identical load tests in different regions and run them simultaneously. 

The final two settings determine the actual load you will be placing on your system. The first is the Virtual User Count. This determines the number of unique virtual machines that will be created for the test. Each VM is equivalent to a user. Importantly, because they are separate VMs, they will have differing IPs, etc. This means they will be handled properly by any load balancer you are running. The second setting is the Test Duration. This gives the total time that the test will run for. Each VM, or “user”, will run the test or project until it completes. Once it has completed it will restart that test. This means that in general each test or project is repeated multiple times by each “user”. 

Sample reports

As we have seen, the Functionize Load Test approach provides a far more realistic to testing your system than traditional systems that just use fake HTTP calls and the like. However, our system also gives far more detailed statistics regarding the system being tested. These include a detailed breakdown of the load applied, performance measurements and a map showing where the tests came from. Below we show some details of the output from a small test run. These figures highlight the level of data you get with Functionize load tests.

In the image below you see the details of the test, how many machines were used, how many times in total the test was run. Any failures or warnings and the actual number of page loads and requests generated by the test. Below that you see a detailed breakdown of the timings for each page loaded as part of the test. 

Further down the screen, you will see graphs illustrating the page performance as the test proceeded. These will allow you to see if there were occasions where load times spiked. They can also help identify if one particular page is imposing excessive load on the system. 

Finally, the Geographical overview allows you to see where the test loads were generated from, thus allowing you to check if the test accurately reflects your expected user base. You can choose to display all the pages that formed the test or pick specific ones. 

Conclusions

As we have seen, load testing is still an essential part of your testing strategy, even if your site is built on scalable cloud infrastructure. Making the testing as realistic as possible by leveraging Functionize’s load testing approach will ensure you understand how your site really performs under load from users. The Functionize solution makes it easy to create load tests by leveraging your existing test cases and using our infrastructure to adapt these into large-scale, geographically-distributed test matrices.