Automating your testing with Functionize

A brief guide for first-time users

Functionize delivers the first autonomous, AI cloud-based testing platform. This guide will help you get started. We’ll show you how to create tests, how to run tests across all major browsers including mobile devices (beta) and how to view the results of the tests. We’ll also explain how Functionize ensures that tests are self-maintaining and explain how things like 1-click updates work. Finally, we’ll briefly touch on API and load testing with Functionize.

NLP test creation

Right from the start, Functionize makes life easy for you. Our NLP (Natural Language Processing) engine means you can simply create your test in plain English, upload it to our system and your test will be created for you. The system understands commands based on keywords (“SCROLL down the page and VERIFY that the buy now button is shown”) as well as natural language (“The top menu bar should be the full width of the page”).

‍

After the test has been generated, you have the chance to approve it or tweak it step by step.

‍

Test execution

The Functionize platform makes it extremely easy to execute tests. You can run individual tests or create orchestrations which run suites of tests, and you can test across multiple browsers and hardware architectures. We partner with Google to provide the Functionize Test Cloud which allows you to test your site from any geographic location and with simulated bandwidth restrictions – essential if you expect your users to come from other countries or to use bandwidth-limited connections.

‍

As the tests run you can see their progress on your dashboard. Once the test has completed you can quickly see if any problems were found. The system also takes a screenshot before, during and after each and every step in the test, which you can scroll through. This will highlight any problems it found to save you having to search for ages, hence helping with fault diagnosis. The visual comparison is done using template recognition, an AI-powered “fuzzy comparison” which won’t trigger a failure just because of a simple CSS style change or rendering artifact.

Test Maintenance

One of the biggest time sinks with automated testing is test maintenance. Traditional Selenium-based systems are dumb and 1-dimensional, being limited to a single selector at a time. The slightest change to a style sheet or a simple change to the page layout breaks the script. Our tests are completely self-healing. Each action is fingerprinted by the system, which uses machine learning to establish what the action is actually doing. So if you restyle your “Buy Now” button as “Add to Cart” the system will cope since it can tell the underlying functionality is the same. Likewise, if you move the button from the bottom left of the page to the bottom right, the system still knows it’s the same button.

Root Cause Analysis

Some test problems are harder to find though. These are typically cases where an action completes without error, you move on several steps in the test, and then the problem appears. Analyzing such failures is one of the hardest things in test maintenance. For instance, imagine your test places an item in the cart, then at a later stage checks if the correct item is in the cart. Imagine that your page has 2 Add to Cart buttons. One for the product being viewed and one for a special offer at the right of the page. If you redesign the page so that the main button changes to Add Now, the system may accidentally select the special offer instead. Now the test will happily proceed, but the failure will become apparent when it finds the wrong item in the cart.

Functionize’s ML Engine includes a Root Cause Analysis feature which is able to deal with this situation. By learning from previous tests, RCA can track-back, find the most likely problems, test the solutions and then present you with the best solutions ranked by probability.

‍

‍

The Functionize CLI

We know that many of you are more at home using the command line than a GUI! Which is why we have developed the Functionize CLI. We’ve built it on top of Java to allow portability across OSes. Simply go to your account, download the CLI zip file to your computer, extract it and you’re good to go. The file will already have your API credentials embedded so there’s no need for any setup. It also includes inline help so if you pass the wrong variables it will tell you. Here are a couple of cool things it can do.

Testing a local server

Often you want to test a feature before you push it to staging. One of the coolest features in our CLI is the ability to test against a local server. Spin up a local instance of your site with the changes you want. Put it on a suitable port and then tell the CLI to use that port for testing. Simply use the start-tunnel command and the CLI will create a secure tunnel:

functionize start-tunnel 5005

Now when you run a test, the Functionize system will run the tests against your local server over this secure tunnel.

Viewing test output

One of the most popular features in Functionize is the ability to view all the screenshots taken during a test. The Functionize CLI supports this using the view-screenshots command.

functionize view-screenshots

The Functionize CLI will launch a browser and take you to a custom URL showing the results of your CLI test. The URL is secure, but it can be shared with other members of your team if they are logged in. This feature is ideal for creating custom test integrations.

Integrating Functionize as part of your CI pipeline

Functionize works seamlessly as part of a continuous integration/continuous deployment (CI/CD) pipeline. For instance, here at Functionize, we use Spinnaker as part of our own pipeline. Spinnaker is ideal because it is open-source and works on IaaS clouds (clouds that run virtual machines like GCP, OpenStack, AWS, VMware, etc.) and container orchestration platforms (clouds that run containers like Docker Swarm, Kubernetes, etc.).

Spinnaker makes it easy to deploy new builds to staging. It can trigger a full set of Functionize tests (e.g. smoke tests, regression tests, API tests, etc.), and if the tests pass, the new code can be automatically deployed using the Promote to Production pipeline. Our customers have also used other CI tools to achieve the same including Travis CI, Heroku and Jenkins. If you want more advice, talk to one of our Solutions Architects.

Other test tools

The platform also provides tools for testing other aspects of your system, not just the web app frontend.

API testing

Included in the Functionize platform is an API testing solution that uses an AI-powered recorder to capture the API test steps. Unlike traditional API testing, our approach tests APIs visually, reducing time and simplifying the process of creating API tests. Our platform also allows you to take responses such as API keys and store them using local variables. They can then be accessed later in the test just by calling the variable. This makes the tests far easier to create, and means an end to repeated copy-pasting of such details. The system also builds in extensive validations and uses machine learning to track how the API response should look. This allows it to be resilient to things like changes in response order.

‍

‍

‍