Learning from its mistakes. How the Functionize AI really works

Functionize is an AI-first company that focuses on test automation. However, we often find that potential customers don’t really understand what AI is or how it actually helps with testing. The problem is, AI is a very broad term that is applied to many systems. However, not all these systems are as intelligent as each other.

Here, we explain how our AI system uses continuous deep learning to build the most accurate possible model of your application. As a result, our tests end up being super-reliable, slashing test maintenance by over 80% and eliminating test debt. But as you will see, this is only possible because our AI learns from its mistakes. And that means it needs to make those mistakes in the first place!

Why AIs are not created equal

Artificial intelligence is defined as a computer system that exhibits the ability to do tasks that would normally require intelligence to achieve. However, that is a pretty broad definition that covers a wide spectrum of possibilities. At one end lie the simple machine learning models that can categorize items into one of several defined categories. At the other end we find systems like DeepMind that can learn how to play complex games well enough to beat world champions.

Narrow vs general AI

AIs can be broadly split into narrow artificial intelligence or general artificial intelligence. This is the factor that determines where they sit on the intelligence spectrum. Narrow AIs are only able to do a single function. For instance, AI-based email spam filters. General AI is much more adaptable. For instance, DeepMind, which can train itself to do many different tasks. Theoretically, there is also artificial superintelligence, but no system really reached that level yet. Broadly speaking, what defines the type of intelligence is how the system actually learns.

Supervised and unsupervised learning

The simplest forms of AI are based on supervised or unsupervised learning. The former is the classical form of machine learning. You use a huge volume of labeled data to teach the computer to recognize a particular pattern or feature. For instance, teaching it to recognize pictures of chairs. This is just like using an ABC book to teach a child to recognize letters. In unsupervised learning, the computer simply looks for interesting patterns and tries to decide if they are significant. Often this is combined with supervised learning if you don’t have a large dataset to start with. The resulting models are forms of narrow AI because they are only capable of one specific task.

Reinforcement learning

More advanced AIs are able to use reinforcement learning. This can be summed up as learning from making mistakes. Each time the computer makes a mistake it has to try again. The next time it may repeat the mistake, but by the third or fourth time it should have learned. This means that over time, it will build a more and more accurate model. As we all know, learning from our mistakes is one of the most powerful ways for a human to learn. Most of us can remember learning to ride a bike. No one starts off able to do this and so you fall off repeatedly. But after repeatedly trying and falling off there comes a magic moment when you learn to keep your balance and actually ride the bike. The resulting models are still narrow AIs, but they are much more adaptable and are getting much closer to general intelligence.

Deep learning

The most advanced forms of AI use deep learning. This is named after the deep artificial neural networks that power it. These can consist of billions of artificial neurons connected in a complex mesh that is designed to replicate the structure of a human brain. These systems are capable of teaching themselves and end up outperforming humans at a range of tasks. However, they are hugely expensive and complex to run, requiring almost unbelievable computing power. These systems also tend to learn from making mistakes. But they are much more capable and flexible than simple reinforcement learning approaches. Such systems definitely exhibit general AI and soon may even reach the level of artificial superintelligence.

How our AI learns your application

Our AI system relies on a number of different learning approaches. At its heart, it combines reinforcement and deep learning to create a detailed model of your application. When you first start using Functionize, the system knows nothing at all about your application. However, it does know about the thousands of applications that it has been trained on over the years. This is rather like us humans. When we first launch a new app, we have no clear idea how it works. But hopefully the app has been designed with a good UI, so we know to expect certain things with its behavior. For instance, we all know that the hamburger icon will open a menu. The icon of a person’s head is likely to be the login button. And so on.

When you create tests using Functionize, we always ask you to add verification steps as you go. These steps are designed to help steer our AI as it tries to learn how your application works. They act as fixed way markers that the AI knows must be there for the test to pass. However, these are not the only things it relies on to determine if a test passed. When you first create a test, you are providing it with lots of additional data about the application. For instance, screenshots for every step, details of all CSS values (including computed values), details of API calls and responses, which elements are visible in the viewport, how long each element took to load, etc.  

Despite this plethora of data, the first time you run this test, the system still doesn’t really know how your application works. There is every likelihood that it will actually fail because it is trying to second-guess what all those values it records should be. Maybe the value is always a bit different or maybe it’s actually a sign there is a bug. For instance, maybe by chance the first time you recorded the test an element took very little time to load (perhaps because you forgot to clear the cache). We could choose to let the system ignore all apparent errors to start with. However, that would mean it carried on making mistakes. It would be like a school teacher always giving pupils a perfect score regardless of what mistakes they made. Instead, we ask you to help teach the AI when it makes a mistake.

Learning from its mistakes

Our system is designed to learn from its mistakes via reinforcement learning. So, each time a test fails incorrectly, you need to report this in the system. This tells it which of the values it is checking matter the most and which ones it can accept a bigger variation for. The next time you run the test, it may still fail. But usually by the third run it will know exactly what can and can’t change each time. At this stage, the model starts to use deep learning to carry on improving its understanding of how your application behaves. This allows it to determine the actual intent behind your test, not just treating the test as a list of instructions to follow blindly.

Benefits of this approach

There are three huge advantages with this approach to learning.

1. The only things you need to do are provide verification steps and alert it when it makes a mistake. By contrast, other AI test automation systems require significant manual setup to teach the system.
2. This approach ensures that the tests will always work on any browser and device. The test simply learns how the application renders differently between browsers and copes with the differences.
3. Using reinforcement learning means our models continuously improve and evolve with your site. Small changes just trigger automatic self healing. Bigger changes trigger Smart Suggestions for fixes. Even if you make a massive application change, it is easy for the system to retrain itself in just a couple of test cycles.  

None of these would be possible if we relied on pure supervised learning as some other systems do.

Seeing is believing

We believe our platform offers the most reliable test automation solution out there. As a result, we can eliminate your test debt and slash maintenance. On top of that, we make it super-easy to create tests, even for people with no prior experience. This is great because it helps further reduce your backlog of test automation. On top of that, we offer integrations with all the main tools, including Jenkins, Jira, Slack, TestRail, and Xray. If this sounds interesting and you want to see it in action, simply book a demo today.