A Brief Overview of Performance Testing Tools

The field of performance testing is undergoing a major transformation. Highly technical tools like JMeter have helped companies make better business decisions in light of understanding a website’s performance across static and dynamic resources.

Since the customer experience online is so critical today, companies cannot afford to be unaware of scenarios and conditions that could bring their site performance to a halt. However, there is room for growth, and Functionize is proud to introduce the next generation of Performance testing and analytics that do not require scripting.

The Metrics That Matter Most when Performance Testing

What are the most important metrics in performance testing? Four Equal and interdependent factors of performance command our immediate attention when testing a web app: confidence, speed, stability, and scalability. Confidence arises from our assurance of an excellent user experience. High marks in the remaining three tend to satisfy the first.

Customers expect a satisfying responsiveness and accuracy. To this goal, performance testing of apps under the stress of high traffic and on various platforms should guarantee both speed and stability. Will the app and its database maintain speed and stability and behave as expected under strain of high volume requests? Furthermore, is the app scalable when traffic far exceeds anticipated levels? These are the benchmarks of performance testing today. How are these crucial factors actually measured? What is the future of performance testing?

Too Many Performance Tools are Required

To measure the performance of a web app, user traffic must be modeled and measured. Commonly the test plan is scripted by QA engineers. If the ideal maximum capacity of a shopping cart app is 100 simultaneous users then required a number of virtual machines is scripted to simulate 100 users plunking away at your user interface. Once rendered, and after all assertions are executed and timed on specified browsers, performance reports are generated. To accomplish this, enterprises now juggle a bevy of developer-level testing tools, all of which realistically require an enormous amount of scripting by QA engineers. Although 100,000 simultaneous users may sound extreme, it is the specified product capability of a tool called Loadstorm.2 In practical terms, if you want to launch an e-commerce web app these may be the typical conditions you need to simulate in testing. Scalability tests then increase users to 200 and report back on how your app works under stress. Performance testing today requires a bewildering array of tools and engineers to deploy them.

To dramatize the point, here is a list of viable competitors in the field today: Apache JMeter, Appvance, CloudTest, Httperf, LoadComplete, LoadImpact, LoadRunner, Loadster, Loadstorm, LoadUI NG, NeoLoad, OpenSTA, QEngine, Rational Performance Tester (IBM,) SmartMeter.io, Testing Anywhere, WAPT, WebLOAD. The list is alphabetized to avoid the spectre of favoritism. Let’s take the first one on the list and vivisect it to see what modern QA engineers are doing at work nowadays.

How to Use Popular Performance Tools like Apache & JMeter

JMeter is an Apache open source tool, probably the most popular in its class for several reasons. Originally designed as a performance testing tool, JMeter now also supports load testing as well as functional testing. Increasingly popular SPAs, Static and dynamic pages are all in the scope of JMeter’s web application testing capability. Here are some of JMeter’s intended test targets:

• Web HTTP and HTTPS
• Web Services XML SOAP REST.
• FTP
• Database JDBC
• LDAP
• Email
• MongoDB
• Java Objects

A wide range of traffic conditions can be simulated, such as heavy load requests on multiple servers, thousands of drone users amounting to hundreds of thousands of HTTP requests holus bolus and at intervals of your design, not to mention database resilience in various network conditions. How do QA engineers actually wield all this power?

The first step with JMeter is to create a test plan, as shown below in diagram 1. And this step really points out the fundamental action potential in JMeter, which is to create an army of simulated users - called a Thread Group in JMeter jargon -  which may all visit your web app UI and execute assertions via HTTP Samplers to see what works and record failures.3 During these assertions JMeter creates reports and logs which you receive as HTML files after the test is complete. 

Diagram 1. JMeter Basic Setup 

With a Thread Group of users configured, the next step is to define the controllers to execute HTTP requests, amounting to the actions or assertions to be executed in your app. If your web app requires authentication, you need to tell JMeter the names of all fields in login form by inspecting the page source or by using the JMeter Proxy Recorder. The concise user manual says it best:

“To do this in JMeter, add an HTTP Request, and set the method to POST. You'll need to know the names of the fields used by the form, and the target page. These can be found out by inspecting the code of the login page. [If this is difficult to do, you can use the JMeter Proxy Recorder to record the login sequence.] Set the path to the target of the submit button. Click the Add button twice and enter the username and password details. Sometimes the login form contains additional hidden fields. These will need to be added as well.”

Advanced test plans are supported as well, including actions such as handling user sessions with URL rewriting, using header managers, building database test plans, and JDBC for defining SQL requests. JMeter also supports development of per-thread cookies. The final component to configure in the test plan is a Listener, which is the element JMeter uses to store results of your JDBC requests and output results post-testing. JMeter creates an impressive set of HTML reports based on test completion. So, does JMeter take home gold, silver, or bronze?

Pleasure & Pain

The items in this list mysteriously jump back and forth between the advantages and disadvantages columns depending on who is reading them. The JMeter UI may be “easy to use” if you’re clear about POST method and HTTP request authentication (the quote from the user manual above may look propitious or ominous depending on your tech background). It may be a headache if you were planning to validate Angular and AJAX scripts; JMeter can’t do this, and you will have to add a tool or a plugin*. For example, JMeter cannot simulate AJAX requests and cannot process .js files. Each AJAX request must be added as HTTP Sampler in JMeter.5 An engineer may love the open source code, but a CFO may see this as a second level development complex, a new HR sinkhole, or a bulging budget for QA now stocked with engineers.

In most top ten lists of performance testing tools, one of the golden featured advantages regularly proclaimed is, “No scripting needed.” But this is only true in the limited case that nothing significant changes in the UI from version to version. Something always changes. The reality today is that QA needs engineers to script testing.

Why JMeter is a Popular Choice

Advantages of JMeter are plentiful and delineate its popularity. You can download the whole shebang free of charge from Apache, and you get the source code in the bundle. You can customize it to your needs and then contribute code back to the developer community.4 The developer community is vast, and there is bountiful support for JMeter online. JMeter is a Java app and runs on any platform. Reporting options are impressive and include charts and graphs. JMeter supports several data formats for report output including JSON, and all popular protocols. Newer versions of JMeter support test plans including load, stress, functional, and distributed testing.

QA Full of Engineers

The only drawback within the intended scope is that JMeter cannot test JavaScript code directly, which means AngularJS, JQuery, Ajax, and all JS libraries. There are workarounds published by the JMeter community, but the app is not a browser and cannot run JS directly. JMeter cannot run JavaScript like a browser does. Anything emerging from JavaScript will require a Sampler or another tool to test. And finally, JMeter only tests web apps; it cannot test desktop apps. The overarching demand for testing tools, in general, is an extraordinary burden arising from the need for testing to keep pace with continuous integration. Accelerated development in Agile and DevOps brought pressure to bear on QA such that many enterprises staff QA with more developers with the title QA Engineer.

Decline of Scripted Testing Apps

Like all testing tools, JMeter has a long list of advantages and disadvantages. JMeter is a tool designed by developers and intended for use by other developers. The provenance of this second level development complex of QA engineers is the success of Agile and DevOps, which together increased development efficiency, and created a bottleneck at QA’s gate. But that was an evolution under pressure, not a design by intelligence. And it was a short term solution which has expired.

‍

Notes 

* Actually there are workarounds for running JavaScript in JMeter using runtime functions, and BSF Samplers. 

References

1. https://www.perftestplus.com/resources
2. https://www.softwaretestinghelp.com
3. Apache JMeter User Manual
4. https://www.linkedin.com/pulse
5. AJAX and HTTP SAMPLERS