Test Ops: What You Need to Know About This Trending Initiative

You’ve all heard of DevOps, where you speed up releases by narrowing the gap between operations and developers. But there’s a new kid on the block.

Test Ops (sometimes called QA Ops) is the logical next step, combining shift right with DevOps to provide real-time deployment and testing.

Here, we explain what Tes Ops is and show why you should embrace it fully.

The history of DevOps

DevOps is a portmanteau of Development and Operations. Put simply, it involves applying Agile methodology to the problems of deploying rapidly and at scale. In the modern world, most software applications are delivered remotely from the cloud. But with this comes a problem.

For instance, if all your company documents are accessed and edited remotely, you become dependent on the reliability of the network and infrastructure. Any outage instantly impacts your productivity. This has become really apparent in recent months, now that so many of us are working from home. Typically, domestic network connections are less robust than office ones. They often have much lower upload speeds, and in many cases, you may be sharing it with your partner and children. But that’s a solvable problem.  

What you can’t solve is issues inside the network itself. Or even worse, issues with Google’s servers. We all saw the impact of that a few months back when Google had problems with their authentication servers. Suddenly, companies across the world found their staff being unable to work. Fortunately, the problems were fixed relatively quickly, thanks to DevOps and Google’s large team of Site Reliability Engineers.

What actually is DevOps

Many people have tried to define DevOps over the years. It’s one of those concepts that people understand but find hard to describe. One attempt at defining DevOps states:

“[DevOps is] a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality.” DevOps: A Software Architect's Perspective. 

In essence, DevOps involves merging the skills of sysadmins and operations engineers with those of developers. That is combined with rigorous testing of software before release, the use of extensive monitoring, and the concept of accountability. The problem is, DevOps is largely reactive. That’s why we need to embrace Test Ops. But first, let’s take a brief detour.

Why testing started to shift right

Testing traditionally happened right at the end of software development, just before release. Over time, we have seen this traditional sequence being eroded. First, we began to shift left, trying to test earlier so as to find bugs quicker. Then we started to see a move to shift right. This involves moving some of your testing into production. In part, this is to support DevOps. Adding testing in production reduces the risk that things will go pear-shaped. For instance, canary testing allows new features to be tested at an increasingly large scale. If you see any instability, you can roll back the release. By contrast, AB testing allows the product team to e.g. choose between two different UIs. And dark launching lets you prepare for a red-letter day launch, giving confidence that your new feature is stable.

From DevOps to Test Ops

DevOps is an essential part of any modern software development cycle. But it has definite limitations. For instance, it is almost always reactive—you monitor the backend and react to any problems that are detected. For another thing, it can only view your system from the inside—you can’t see issues that only impact the frontend. Test Ops attempts to address these issues.

Test Ops refers to the use of QA test automation approaches to monitor the real-time performance of your production environment.

Active, not reactive

Test Ops relies on active testing of your system to identify problems before they occur. Typically, that means using your automated tests to test your production system. This sounds easy, but of course, most automated tests are designed to run in your local test environment. That means cloud-based testing is a prerequisite for Test Ops.

Testing with real use cases

The second aspect of Test Ops is related to the first. By using your automated tests, you ensure that you are testing real use cases. In other words, you are seeing how the system reacts from a user’s own perspective. This is especially important for testing complex flows that interact with multiple backend elements. For instance, logging in interacts with AAA, pulls the profile from the DB, and serves the homepage from your frontend engine.

Identifying errors DevOps can’t see

That use of real use cases also achieves the third element of Test Ops. Often, errors are only visible to the actual user. For instance, seeing that the login is much too slow, or that the wrong element has been displayed on the screen. These errors are effectively invisible to the backend. This is especially true when you are dealing with in-network caching, where some or all your content is delivered from a CDN.

All these features reflect the use of traditional QA techniques in an operational environment, hence the term, Test Ops.

Test Ops is a two-way process

Defining test plans is something of an art. You might want to test every possible scenario and user interaction, but that’s often not possible with the time and resources you have available. As a result, even the biggest companies in the world can suffer from bugs. What Test Ops enables is a two-way flow of information about which features your users actually access and how they do it. Moreover, it can reveal issues with how you actually run your testing. For instance:

• Your testers may always be accessing the application over WiFi. But if your user base is primarily using cellular data, they will see a very different experience. Your DevOps and support teams can help provide this sort of usage insight. You can then adjust your testing accordingly to make sure you take that into account.
• What if your testers are based in Ukraine, but your user base is in the US and South America? Most applications now employ localization and internationalization. These are notoriously hard to test because they depend on so many variables—the IP address, language settings, cookies, etc. If your testers know where your customer base is, they can use the Functionize Test Cloud to allow them to run tests from different locations.
• Modern applications are often completely dynamic—each user sees a unique view depending on their profile. Your application may even offer customized views. It is near-impossible to know in advance how users will navigate such an application. That means your testers and product team are taking educated guesses as to which user flows to test. This means you need to monitor how users really interact with your application in production.

All these are cases where your testers need to see what is happening in production.

How Functionize delivers Test Ops

Functionize is the ideal platform for Test Ops. We use AI to deliver smart test automation from the cloud. This delivers 4 key benefits:

1. Creating tests is faster and simpler. We offer multiple approaches for test creation, ranging from Architect, our ML-powered test recorder to NLP test creation. We even allow you to specify tests based on how real end users are interacting with your system. The upshot is, anyone can create fully-functional automated tests in minutes while retaining the advanced functionality of classical test scripts.
2. Tests need no maintenance. As you define tests, our platform creates complex AI models of your application. Our tests can self-heal, so they aren’t affected by things like UI redesigns or simple changes to the application logic. Even if something big changes, SmartFix suggests the most likely fix and allows you to apply it in one click.
3. Visual testing checks the entire UI. Test scripts have one major flaw—by definition, they only test the specific elements you tell them to. A modern UI can include thousands of elements on a single screen. It is simply impossible to test all these with traditional test scripts. Instead, we rely on a visual testing approach. This means we compare every test run with previous runs. If the system sees that some element has changed more significantly than usual, it will flag it to you in a screenshot. This even works for computed CSS elements.
4. We allow you to add tags to monitor how your users really interact with your application. This often reveals unexpected user flows or unplanned ways of accessing certain screens. Once you see this, it is easy to identify gaps in your testing. Moreover, you can use these flows to define new tests.

All these features are important for Test Ops.

Test Ops in practice

So, how do you actually go about implementing Test Ops with Functionize? The first step is to automate as many of your tests as possible. Focus on tests that relate to the most frequent user flows in your system. Next, create a comprehensive set of tests that reflect the typical use of your application. Effectively, this is a form of smoke test—your aim is to spot any problems before your users do. Now, use the Functionize Test Cloud to conduct tests against your production system from every region where you have users. Finally, monitor the live test results and trigger alerts if anything is wrong. You can rely on our advanced visual testing to flag content errors or use our visual completion metric to spot when the backend is running slow. To see how easy all this is, sign up for a free trial today. Or you can book a live demo with one of our team.