Functionize Root Cause Analysis & Self-Heal

Overview

One of the biggest costs involved in testing is maintaining tests that fail, due to changes over time in the system being tested.

A major factor in this cost of test maintenance is that when a test fails, the failing action is often not the root cause of the failure. The root cause may have occurred several steps earlier.  

A typical example is when changes on a page cause an incorrect button to be selected and clicked. The resulting page may not be the expected one, causing a subsequent action to fail. 

Identifying such problems can be time-consuming, since it requires looking back through the actions that led to the failure, in order to identify the root cause. Successful root cause identification often requires a detailed investigation, including examination of the values and attributes of fields, and examining and comparing screenshots. 

Having identified a possible root cause, the test must be modified and re-run to verify whether the cause was correctly identified and addressed. 

Functionize Root Cause Analysis and Self-Heal helps automate this process, reducing the time and costs involved in this important test maintenance activity. This is achieved via three features that work together to identify and help resolve test failures:

1. Root Cause Identification: Automatically identifying likely root causes for a failure.
2. Smart Suggestions: Suggesting possible fixes for the failure. If the user accepts a suggestion, the test is updated automatically.
3. Self-Heal: Automatically resolving the failure, by:

• Automatically testing suggested fixes and reporting on the results
• Automatically implementing an approved suggested fix

Each of the above features is described in more detail in the following

Root Cause Identification

Identifying the root cause of a failure is at least half the battle in correcting a failed test, and by itself can save significant time. Root cause identification is explained in more detail in the “How it Works” section below.

Smart Suggestions

Once possible root causes have been identified, suggestions for resolving the failure are automatically generated. These suggestions save time by identifying possible resolutions and allowing the user to update the test accordingly. Accepting a suggestion will update the test. Smart suggestions also support the self-heal feature described below.

Self-Heal

Each potential root cause can have several possible resolutions. In cases where the correct resolution is not immediately obvious, the self-heal feature can be used to automatically re-run the test with each of the available suggestions. The user is then given the option to approve any successful resolutions to the problem, and the test will be automatically updated with that resolution.

Learn more

How it Works

Root Cause Analysis can be a complex task, and Functionize brings several technologies and techniques to bear in order to address it.

Historical Data

Historical data about previous successful tests is a key factor used to identify root causes. Comparing a failed test run to previous successful runs allows anomalies in normal test behavior to be identified. These comparisons are performed using multiple techniques, described in the following sections.

Rule-Based Expert System

A rule-based expert system is used to identify many common failure scenarios, such as various kinds of element selection failures, or overly-specific comparisons that fail when data changes.

Each action in the test is examined and tested against a set of rules, compared to related actions earlier in the test, and to successful actions on previous tests.  

An intelligent scoring mechanism is used to evaluate the confidence in the selection of elements and in the correctness of actions executed. These scores are used in assessing the correctness of each action, to help identify root causes and possible solutions. The scores are generated by the machine learning algorithms described below.

Machine Learning

Machine learning is used in root cause analysis in multiple ways.

ML models are used to generate scores characterizing the confidence level in element selection and action correctness. These scores are used by the expert system to help detect failing actions. 

ML models are also used to identify root causes by learning what successful actions look like over time, and using this training to identify failed actions. This allows the system to adapt to changing application behavior, and more effectively identify the actual root causes even in cases where the expert system cannot.

Learn more about ML engine

Examples

Element Selection

The root cause of many test failures is element selection - for various reasons, a test may identify and select the wrong page element, resulting in a failure. This failure may occur immediately or on a subsequent action.

Root cause analysis can identify the original failing selection, saving time in debugging the cause of the test failure. Smart suggestions can suggest changes needed to select the correct element. Self-heal can implement and validate these changes.

Comparisons

It is common for a test to rely on a comparison that later fails due to changed content in the element being compared. Root cause analysis can identify these cases and suggest fixes, such as using less strict comparison operators - e.g. ‘contains’ instead of ‘equals’, or using substring comparisons.

Future Scenarios

Root cause analysis will be able to detect the following scenarios in future versions.

Slow loading

Tests can fail due to slowness when loading a request. Functionize’s monitoring of action durations allows it to compare to historical behavior and identify cases where an action fails due to slowness in loading. In this case, the suggested fix will be to either add a delay or simply re-run the test. The self-heal feature can automatically implement and validate the adding of a delay.

Failed loading

Request failures, such as HTTP errors (4xx, 5xx), can lead to failed tests. Root cause analysis can identify the action which resulted in a request failure.

Failed authentication or authorization

Root cause analysis can detect scenarios where authentication or authorization fails.