Functionize as a new user 3: Viewing Test Health Reports

This is the third part of my blog series exploring Functionize as a new user. Here, I look at the tricky subject of test health and test maintenance.

How did I get here?

As I explained already, I have been developing a web-based application to help users test and simulate robots. This application is based on robotbenchmark. In my journey I realized that web applications need to be tested, and that my old-fashioned approach of printing debug statements to the console wasn’t going to be good enough. Fortunately, I came across Functionize and was amazed at how easy it makes test automation.

In my last post, I showed how the Functionize platform provides two ways of visualizing test results. With the Slider View, I can watch a “manual gif” of my application during testing. I can see the site before and after each test, and compare the two runs side-by-side to track what test was being executed when.  

I explained how the Functionize Slider View makes it extremely easy to analyze test results. I was able to identify a key flaw in my test design and rectify it in minutes. My mistakes in test creation segue nicely into another feature of the Functionize Platform. In addition to the Slider View, the Functionize platform provides Test Health Reports. The Slider View lets me analyze the results of my tests created in Architect. The Test Health Report lets me analyze the performance of my tests themselves.

How to understand the health of my tests

Although I’m new to the world of test automation, I am not new to the world of software development. So, I know that any and all software can be buggy or unstable. Fortunately, Functionize makes it really easy to see whether my tests are working well. The Test Health Reports gives a concise report containing tons of useful information. 

This information is not only useful for me. There’s a lot of crucial information in here I might need to share across my team as well. Imagine if I were working with a bigger test team, and my login test had failed. Everyone else’s tests downstream would have failed, seemingly without reason. The Test Health Report, however, captures failures like this visually. My Test Health Report is below. 

The Test Health Report shows me how many tests I’ve created for my project, what kind of tests they are (Architect or NLP), when the tests have executed, and the test results. Furthermore, the Test Health Report will show a Browser Health history. Mine is pictured below. 

Like with test health, this report shows me my results by browser. As you can see, robotbenchmark runs primarily in Chrome, so Chrome is the only browser I tested. That said, Functionize is capable of testing Safari, Firefox, and IE, so I can easily open my project to additional customers in the future.

Drilling in deeper

The last feature of the Test Health Report I will discuss is the test case history report. This graph shows me when each test case was created. Using the Test Health Report, my entire test team can see what test cases were created and when. For my testing, this is especially useful because it shows my creation of new tests to replace my failed ones. 

Test Health Reports give me a visual report on my tests themselves. These reports give me the power to visualize not just my test results and history but also my test strategy. I can understand if some tests are taking too long, or if two tests interfere with each other. I can view results by browser and look for incompatibilities and inefficiencies. 

The Test Health Report streamlined my testing. Moving deeper into my project, the Test Health Report also opened up my project to other testers. My team will be able to jump into testing quickly and easily due to this clear overview of my testing plans.

Self Healing

I was talking to a friend about my experiences with Functionize. She regularly used Selenium for test automation and asked me if I had any problems with test maintenance. I asked her to explain what she meant. Apparently, when you change your app or site, Selenium tests typically break, giving lots of false positives. When I started to read more about this on the Functionize site, I found that they use machine learning to avoid this issue. I read about Self Healing tests and was intrigued. So, I decided to make a slight change to my login page and see if I could trigger a test failure.

I knew from my friend that Selenium gets easily confused if buttons change on screen. So, I changed the “Log In” button to “Login” and swapped it with the “Sign Up” button. Instead of failing, my test passed. However, when I looked at the detailed steps for my sign up test, I noticed that there was a little label by one of the steps. This was informing me that the test had self healed. This was really comforting to me. It means in future I will be able to see if any of my tests have healed themselves.

Conclusions

Functionize has really helped me get started with test automation. I’m still a beginner, and yet I feel like I am able to approach testing with confidence. Functionize doesn’t just show me the results of my tests, it also lets me see data about their health. These reports showed me the status of my tests themselves. The Test Health Report and Browser Health Report gave me the means to view the status and history of every test across the project. 

In this article, I walked you through my experience analyzing my test results. In my next and final post, I will explore exactly how Architect uses ML to give me the full-featured tool for my software QA testing. To follow along with your own free trial, visit https://www.functionize.com/free-trial  today.