Test automation strategies: If it ain’t broke, don’t fix it! Right?

The common reaction to organizational change is “If it ain’t broke, don’t fix it!” This is especially true in the world of test automation. What makes it trickier is that actually some things aren’t suitable for transitioning—it’s not as simple as “just automate every test”.

Here, we look at this conundrum that faces every company embarking on modernizing their testing and explain how to decide your strategy.

Testers are human

Testers are humans just like the rest of us. They have spent years building and optimizing your test strategy. They are rightly proud of what they have done. This often lies at the root of the mantra “if it ain’t broke, don’t fix it”. It’s closely aligned with risk aversion and wanting to make safe choices. This is partly down to their small-c conservatism. But it’s also because, almost by definition, testers are careful, methodical, risk averse, and even fussy.

This is by no means a bad thing—you really wouldn’t want your testers taking risks about releasing buggy software! The problem comes when you look to update your testing practices. Here, you need your cautious QEs to take what they believe is a risk. Namely, rely on a new tool to do their testing for them. This would be a completely appropriate position if their existing tool was perfect. However, in our experience, no test team has perfected their test automation, partly because perfect automation is something of a pipe-dream. So, don't try to recreate the wheel by ripping out your entire test strategy and replacing it with a new one. Instead, evaluate what's working and not working to identify areas for improvement.

Know your current automation tools

Before you can make a reasoned decision on whether to adopt a new approach to testing, you need to really understand your current situation. The majority of test automation tools and frameworks are based on test scripts. Test scripts are perfect for certain types of testing and were one of the first universal approaches to test automation. The fact they have endured almost two decades shows that they work. The problem is, they lead QA teams to lose sight of their core aim, namely testing the reliability of your software. Instead, the focus becomes creating and maintaining the test scripts. Two things in particular cause issues:

Scripts are fragile. Test scripts are created from repeating a simple 3 step process. First, find and select some element in the UI like a button or form field. Second, perform some action on that element, such as clicking it. Third, evaluate the outcome and see if it is what was expected. These are the exact same steps a human tester would do. The problem is, step 1 is susceptible to changes in the UI and site logic. If you change where a button is on the page, the script might not find it. In fact, even quite minor changes end up having a big impact, resulting in the QA team having to update all their test scripts. 

Scripts are limited. By definition, a script only checks what it is told to check. This is where test scripts differ from human testers. A human tester will instantly spot if the UI is messed up. But the test script is only looking for the specific elements it has been told about. If half the images don’t load, the script won’t care. Of course, you can tell the script to check each and every element, but that becomes incredibly tedious and slows everything down. 

Despite these problems, there are certain things where test scripts excel. Not least because a script is an actual software program. This means you can build quite complex logic into your test scripts. However, if your test scripts are taking a lot of time to maintain or you anticipate your UI will change frequently, it may be worth looking at smarter automation approaches.

Adding Functionize to the mix

The Functionize platform is a smart test automation solution that leverages AI to solve some of the issues above. Specifically, we have designed it to achieve three things:

Simplify test automation. Test automation is complex and time-consuming to set up. Writing test scripts is slow and painstaking, often requiring people with both developer and QA skills. Our approach is to remove that complexity and allow anyone to create automated tests. Architect, our advanced test recorder, allows you to create tests simply by interacting with the UI. Under the hood, it’s recording a huge volume of data and building a model of your application. Architect allows you to test things like 2FA login and also offers a number of advanced features, such as test data management.  

Test the whole UI. Functionize records screenshots before, during and after each test step. We also check other aspects, like computed CSS values. The system then compares these with what it expects to see, based on previous successful tests. Our system flags anything out of the ordinary and highlights it on the screenshot. The result? The whole of your UI is being checked each time you run a test. 

Allow the team to focus on testing. We believe your test team should be free to focus their attention on testing your product. They should be checking your software, developing tests, and tracking down bugs. So, Functionize tests are maintenance-free. Or to put it another way, you create the test and it just works. This is possible because we use dynamic element recognition to choose elements in the UI. If your UI changes, our system dynamically heals and gets on with the test. That frees up a significant chunk of test engineers’ time.

Manual testing isn’t going away

Test automation allows testing to be run continuously. In turn, that helps you release faster. However, certain types of tests simply cannot be automated using conventional tools. And some other tests simply aren’t worth automating. This is why manual testing isn’t ever going to vanish. Let’s look at a couple of examples to see why.

Testing modern signup flows. Signing up to websites used to be easy. Enter your email, choose a password that passed the password-strength check and that was all. Then things started to get more complex. For starters, many countries now require the user to click on a confirmation email before an account is created. Then there’s the whole issue of two-factor authentication (2FA). Both these are something of a nightmare for test automation. How can your test script check for an email or SMS sent to a different system? The usual solution is to keep this testing entirely manual. 

Tracking down a bug. Testing isn’t just about finding a bug. It’s also essential to find out how to recreate it. That way, the developers have clear data they can use to analyze and fix the bug. Typically, this involves a combination of experience and repeatedly trying different things to see what conditions trigger the bug. This sort of exploratory testing is really hard to automate effectively. This is because a human could complete dozens of iterations of testing before you even finished creating an automated test. As a result, this sort of unplanned or ad hoc testing is always best done manually. 

But manual tests can only go so far, since you're limited by the size of your team and the available time for testing. Modern testing solutions like Functionize allow you to automate signup flows and other tricky scenarios, like visual image comparisons, that usually require a manual tester. This sort of hybrid testing strategy can bring real gains.

Choosing an automation strategy

As you may have gathered, any successful test strategy will include a mix of manual and automated testing. The following flow chart will help you understand what option is best for any given test.

Review your strategy regularly

In an ideal world, your test strategy should evolve continuously. It’s really important you revisit your strategy and review it regularly. Specifically, you should look at three things:

• Identify any test scripts that are out of date. Then decide whether these should be renewed or transitioned to a modern tool like Functionize.
• Monitor which test scripts need constant maintenance and upkeep. These scripts are ripe for transitioning to a low-maintenance solution like ours.
• Find any manual tests that are being executed regularly. Chances are, that means they are ready for automation.

Conclusions

As we’ve seen, people frequently react to organizational change with the phrase: “If it ain’t broke, don’t fix it!”. This is really comforting when you don’t realise how broken your process is. All too often, this is an excuse to not modernize or update the way they do things. And frequently, they really don’t realize that it is broken. By following the strategy above, you will be able to steadily reduce your automation debt and optimize your whole testing strategy. To find out how Functionize can help with this journey, book a demo today.