The Functionize manifesto for modern test automation

Some years ago, we introduced the Functionize testing manifesto. This manifesto has informed all our product and technology decisions. Recently, we updated it to reflect how our products have matured over the years.

Quality is in the DNA of every great company

There’s an old adage “quality over quantity”. Good developers have always known this to be true. There comes a time in any software development cycle when the developers start to get proud of how much code they are deleting. At this stage, the developers are aiming to create the highest quality and most efficient code possible. Good product managers also know this. Feature bloat, where new features are added just because they can, is a sign of a poorly designed app. A good quality app will do a few things but do them extremely well.

This suggests that quality should be an integral part of any successful company’s DNA. For software, quality is achieved through testing your products again and again. Ensuring you never allow any regressions to slip through. Testing every new feature and keeping on testing right through to production. 

People often ape Google, even when that is inappropriate due to the difference in scale. But when it comes to testing, they definitely have the right idea. Google was one of the early adopters of Selenium because they were serious about quality. They still are serious, and now they have moved well beyond Selenium. We believe every company should take a leaf out of the Google book when it comes to testing. Achieving high quality isn’t about employing more and more QA and test engineers. It’s about using your test engineers as your in-house test consultants, supported by an intelligent test platform.  If you do this properly, quality will be baked into all your products.

Software testing drives revenue

For years, testing has been the poor relative in the development process. Companies know they need to test, but management often resents testing as it looks like poor ROI. One of the problems is that it’s extremely hard to measure the success of testing. If your tests find lots of bugs it doesn’t look like a success for the testing, it looks like a failure for the developers. If you don’t find any bugs it can look like it was an unnecessary expenditure of effort.

When you widen the focus of testing to cover the whole product life cycle, it changes your perspective on how to measure ROI. Suddenly, testing becomes a key driver for revenue. It becomes the oil in the gears of the development engine, speeding up delivery and reducing friction. It becomes the canary in the production coal mine, warning of problems before they hit your revenue. And it becomes a key driver for responding to customer demand, helping drive up user numbers and revenue. Above all, it becomes the measure of how well your software is performing and developing. 

Our simple message to management is, stop thinking of testing as a money sink. Start to view it for what it is, the defender of your success and revenues.

Intelligent testing is your brand’s shield and protector

Buggy software is a nightmare for your reputation. Even big companies with loyal fans find their reputations suffering if they let a bad bug slip through. So, ask yourself: what is your brand and reputation worth to you?

The problem is, even a small application quickly becomes too complex to test everything manually. This complexity skyrockets when you add in things like responsive UIs, cross-browser testing, multiple OSes, and user customization. The traditional meaning of test coverage represented how much of your code is covered by tests. But we think a more useful definition is a combination of this multiplied by how many of the possible combinations listed above you actually test. 

The only solution for achieving higher test coverage is test automation. But normal test automation suffers from many of the scaling issues of manual testing. Every browser and screen resolution combination may need the script to be tweaked. Many features may not be testable at all. And if your application allows the user to customize their home screen, etc. it could be impossible to test all the combinations. 

So, we believe the real answer is intelligent testing. Write the test once, and it will work across any combination of device, screen resolution, OS, etc. Adopt intelligent testing and you can be sure that your true test coverage will increase enormously. 

In short, intelligent testing becomes the shield and defender of your brand and reputation. It is almost impossible to put a price on this. 

Intelligent automation just works

Test automation revolutionized the software world. It speeded up testing and thus application development took off. It allowed more thorough testing, leading to an improvement in software quality. Above all, it allowed us to cope with the explosion of smart devices and web apps over the past decade.

But classic test automation is dumb. And, like all dumb systems, that limits its growth and utility. Test maintenance becomes a dominant part of a tester’s time. Test scripts need to be rewritten for every browser or device, slowing down the speed of test creation. And tests can only actually check a small percentage of your application. 

Testing should be intelligent. Put simply, (well designed) intelligent automation just works. Intelligent automation means an end to test maintenance, simpler test creation, and the ability to test everything visually.

Tests should never break

The advent of test automation saw the advent of a new problem, test maintenance. Automated tests rely on identifying elements within the UI in order to interact with them. The problem is, as products develop, the UI changes too. Sometimes these changes may be small, such as restyling buttons to gain a wider border. Other times they may be more significant, such as redesigning the whole UI. Either way, they probably break your traditional test scripts.

The upshot is, test churn and test maintenance, where tests have to be constantly updated and rewritten. This time spent on test maintenance is time lost. Tracing and resolving these issues is a massive time sink for your skilled test automation engineers. As a result, it becomes a waste of a valuable resource. 

We believe things can and should be different. Tests shouldn’t be so brittle that they break from release to release. Instead, the system should apply intelligence to learn what the test is actually doing. In other words, it should behave more like a human would. When your favorite app gets a redesign, you just look at the new UI and find the button you wanted to tap. The same thing should be true for test automation. Now your engineers can concentrate on doing their job, rather than just wasting time fixing things that shouldn’t break. 

Shifting left is not enough. Shift right

Originally software testing happened just before release. This seemed logical and reflected how software development sought to copy other engineering projects. Then people realized the benefits of shifting left. By shifting testing earlier in the design process, you find the bugs earlier, and can fix them more effectively. In its ultimate form, this leads to methodologies like Test-Driven Development.

But we argue shift left isn’t enough. Testing needs to be shifted right as well. What do we mean by that? Well, testing shouldn’t stop the moment your code ships to production. Often, you will only really understand how your code performs once it is being used for real. Of course, companies have always responded to bug reports from users. But this unstructured production testing is not efficient. And it looks bad when your customers find the bugs, not you! Instead, you should be looking to test in production. 

This requires three key elements:

1. Instrument everything. Ideally, make sure you can track the exact way users interact with your application. That way you can check if you really are testing the right things.
2. Use dark launching, feature flags and canary testing to make sure you have control over all your new feature launches. That way if something goes wrong, you can see it and fix it before it becomes a big problem.
3. Apply the principles of continuous quality and continuous experience monitoring. This means using your automated test platform to check how well your production system is running.

Shift right and your users will become your most powerful test tool.

Agile DevOps requires continuous quality

DevOps is critical to the success of many companies. Without DevOps, many services end-users take for granted would simply fail. At its heart, DevOps is part of the Agile world. DevOps straddles the border between developing software and running it in production. It is what you get when you apply the Agile approach to development to services in production. And, of course, that is exactly the interface where testing traditionally sits.

DevOps teams will benefit from continuous quality and intelligent test automation. Continuous quality means systems that constantly test your production site from the end user’s perspective. Any problems can be immediately notified to DevOps who can then fix them. This is backend monitoring on steroids.

Testing must be cloud-native

For far too long, testing has been run in-house using old servers, begged resources, and outdated infrastructure. We believe testing should be run in the cloud, be it a public cloud or your private cloud. The cloud allows you to leverage an almost infinite resource pool. Testing done like this will be faster, more efficient, and cheaper. Testing deserves just as good tools as the rest of the development team.

The arguments for testing in the cloud are the same as those for moving your production servers to the cloud.

1. It’s cheaper. No need to supply and maintain your own hardware. You also only pay for resources you actually use.
2. It’s scalable. Cloud offers the chance to run tens of thousands of tests in parallel. Almost no companies can test in-house at this sort of scale.
3. It’s reliable. Virtual servers are far more resilient than physical servers. This means you won’t need to worry about server failures or downtime.

Without the cloud, you can’t achieve continuous testing. And there’s one other benefit that you can’t get with your own infrastructure. Cloud testing allows you to run tests from anywhere in the world. In turn, that means that your testing is far more realistic, allowing real localization and internationalization testing!

Testing must be strategic. Don’t leave it as an afterthought

It’s all too easy for management to overlook the importance of testing. That is, right up to the moment things go wrong. At which point the test team becomes the target of everyone’s ire. The problem with this view is it is rather short-sighted. If your test team failed to spot a bug it probably isn’t a reflection on their performance. It is more likely to be a reflection on your lack of strategic focus on testing and quality management.

In our opinion, QA should be a C-suite responsibility. No one laughs at the idea of a Chief Compliance Officer, so maybe you should consider appointing a Chief Quality officer? Successful companies are the ones that adopt the mantra of test early, test often, and test in depth. This means testing becomes a key strategic requirement in your organization. If testing needs more resources, make them available. If there is tension between your test and dev teams, try merging them. If you want to release new features frequently, look into continuous quality. 

Intelligent test automation must be at the heart of your approach. We won’t pretend it can solve everything – sometimes you still need manual testing. But, used strategically, it can save you from a lot of heartache and pain. It shouldn’t be just an afterthought, it should be one of the first things you think of when planning a new product.

Read full manifesto.