Case study: Functionize Eliminates Agvance Test Maintenance with Machine Learning

Background

Through their Agvance product line, Software Solutions Integrated, LLC (SSI) has become one of the most well-respected providers of integrated software to retailers in the agriculture sector. Agvance has been developed over a number of years and provides retailers with focused solutions for agronomy, grain management, energy management, and accounting. In early 2017 the company decided to switch their focus to developing a modern cloud-based web application, called Agvance SKY. This is completely replacing their legacy desktop Windows application suite.

Challenges

Before moving to a cloud-based web app, Agvance had spent some years developing a very large end-to-end test suite based on a well-known competitor. This suite gave them the assurance that any unexpected bugs that crept into the code would be identified prior to release. But as Todd Erickson, their Director of Technology puts it “One problem with our old legacy tests was that they tended to be brittle.” He described the pain caused by moving their app from Windows Vista to Windows 7, which caused tests to fail because buttons moved slightly on the screen.

When Agvance decided to build their new cloud-based app, they decided from the start to move to developer-driven testing using CI-CD. They needed a testing tool that could provide them with the same safety net they had before, but one that would be much less brittle. In particular, the tool had to be able to cope with the responsive nature of the app. In other words, they needed tests that wouldn't break just because the screen was resized or the page restyled. Particularly important, they needed to be able to develop the tests while the app was being developed. This was simply not possible with their previous setup since the smallest change to a screen would cause all the tests to break.

How Functionize Helped

After hunting for some time, Todd came across one of our blog posts and decided to explore whether Functionize could help. After seeing a demo, Agvance signed up with us 6 months ago. From the start, they were impressed by how easy it was to create tests. Functionize tests can be created several ways, including a simple, intuitive, point and click interface, whereby a user trains AI models to understand any web application. This means pretty much anyone can create tests. As Todd put it “Being able to take someone without strong developer skills and be able to have them develop valuable tests have been really good.” This is in marked contrast to their previous experiences where creating tests needed detailed scripts and developer expertise.

One of the biggest issues they faced when looking for a testing solution is that Agvance SKY is based on the Angular JS Framework. This framework uses a dynamic DOM element and relies on state management to enable intelligent form submissions. Creating automated tests with this framework is almost impossible for standard tools such as Selenium. However, Functionize was able to effortlessly handle these complex scenarios by using its inbuilt smart DOM selection logic. This meant that an exhaustive test suite could be created without the need for complex workarounds and code hacks. 

Agvance is also really pleased with how robust Functionize tests are. Because our platform uses AI and machine learning to identify elements within the app, tests are no longer brittle. This means that elements such as buttons are no longer simply defined by their position on the screen, instead, they are defined according to their actual functionality, regardless of where they are on the screen or how they are styled. This means Functionize tests work in the same way as a skilled manual tester would.  

Todd again: “One day we had restyled a page pretty drastically. The head of QA came to me and said she couldn’t believe it. None of the tests had failed. Functionize was able to recognize what each element on the screen was and was able to recognize that there was still good data coming back.”

Benefits & Outcomes

The biggest benefit Agvance has seen since moving to using Functionize is the fact that tests remain good for the lifetime of the feature they are testing. As Todd puts it “All the effort the QA team puts into writing tests is never lost.” This means that the QA team can spend more of their time on creating new tests and less on maintaining brittle old tests. Furthermore, it means that they have been able to reach the same level of test coverage they had for their legacy app, but in a tiny fraction of the time. This means they again have the feeling of having a safety blanket, knowing that if a new feature unexpectedly breaks something it will trigger an automatic test failure.

Another direct benefit of this is that testing is now completely scalable. With their legacy application, certain elements still had to be tested manually, and any change to the UI required the tests to be re-written, a process that required developer input. Now all those tests can be automated, and almost anyone on the team can create new tests and add them to the suite. This opens up a path to embedding QA engineers directly into each of the developer teams, so they can sit alongside and streamline the process even further.  

When he spoke to us Todd was excited to learn that Functionize offers full integration with build tools such as Jenkins. This means Agvance will be able to integrate the tests into their CI-CD workflow. So long as all Functionize tests pass, Jenkins can be set to automatically push a new version from staging to production.

What’s Next

So where will Agvance go next with Functionize? Well in addition to exploring integration with Jenkins, Todd wants to start using the automated tests in a more intelligent fashion. Rather than just viewing tests as pass/fail he intends to use Functionize’s analytics to monitor the performance of the tests. If a test suddenly starts taking twice as long to complete this could point to an underlying problem in the codebase. Todd is also really keen to explore Functionize’s most exciting new feature yet – our patent-pending AI technology that uses natural language processing (NLP) to directly convert test plans and user journeys written in plain English into robust, ready-to-use automated tests.