The Pursuit of Continuous Testing

In a recent Functionize article, I wrote about the need for continuous testing, how CT differs from testing automation, and the importance of continuous assessment and risk mitigation while exploring how to implement CT. This article is a follow-up, in which we’ll look at the CT challenges, scope, benefits and best-practices.

Look both ways: The challenges of continuous testing

Analyzing continuous, automated testing can be very challenging. A serious challenge that we see with testing tool customers is that they run all this automated testing at a very high pace and a high output volume—but it’s quite difficult to analyze it all and decide that it’s done-done. As human beings, it is very difficult to try and absorb all the output from multiple testing tools, distinguish black-box from white-box from functional testing from regression, perform static analysis and dynamic analysis, and ensure adequate code coverage.

Ensuring that both developers and operations teams continue to have clear visibility into  testing analytics is challenging—yet it’s essential. As much as practicable, continuous testing seeks to shift the root of the testing vector further leftward—further upstream—to give as much incentive as possible for all developers to get near-immediate feedback.  

But CT also requires looking further right—much further downstream. Of course, the testing team take serious interest in how features are holding up in the development and test environments. But it’s also very important to continually observe how users perceive the feature, how often it is used, and how it actually behaves in the wild. These observations will actually provide more valuable insight into the impact of the testing efforts. 

Think about it. If a product release contains building features that only a few people are using, then the testing and risk mitigation is rather easy. For features that are put into use by 80% of your customers, then those features are business-essential—which means more testing scrutiny and more concern for testing risk assessments.  

Another challenge is how to be smarter. Most informed teams really don’t need to run more tests. They need to test smarter. When, for example, you think about your noisy tests—such as a suite on which it’s necessary to spend three hours analyzing it—you are probably ignoring most of the results. Any test that your team ignores should be removed. Why? For the simple reason that you’ll shift a bit more of your attention to other variances—which is a very, very good thing!

CT goes well beyond test automation

Drawing a distinction between test automation and continuous testing may seem like an exercise in semantics, but the gap between automating functional tests and executing a continuous testing process is substantial.

Any true change initiative requires the alignment of people, processes, and technology—with technology being an enabler and not the silver bullet. Yet there are some basic technology themes that you must explore if you want to move toward CT. In general, it is crucial to shift focus from mere test automation to automating the process of assessing and measuring risk for an planned product release.

Don’t let tools sidetrack you away from CT

Even a testing team that achieves a decent level of success with conventional automation tools can encounter a major roadblock when their management moves to adopt modern architectures and delivery methods.

No technology or tool will ever give you out-of-the-box continuous testing. Like DevOps and Agile process improvements, CT requires that people, technology, and processes undergo change. But it can be excruciatingly difficult to attempt such changes in people and processes when your technology is inadequate. 

Assuming that a large and diverse tool set can be highly beneficial, it’s important to be willing to move away from the comfort you might have in depending on a suite of tools from a particular vendor—that doubtless has a particular set of strengths and weaknesses. In most environments, it is much better to distribute a broad array of SDLC sensors all throughout the development pipeline. To optimize both the value and the accuracy of such sensors, it’s essential to deploy the sensors with great care—not in the commonly ad hoc manner of too many product development teams. Ensure they are configured and applied consistently, and that the sensor output funnels into an AI engine that will cross-verify and correlate all observations from all tools, and across multiple test runs. Not only does this approach greatly enhance the ability to identify application hot spots, but also minimizes the risk of encountering false negatives. 

Benefits of continuous testing

Continuous testing is a test-early, test-first approach, in which you get much better coverage and confirmation that new product features are functioning according to design. You can also readily see which changes don’t pass preliminary checkpoints. Also, live-service monitoring can help in spotting and condition changes that might affect quality or risk.

There are many other benefits that your team can gain from implementing continuous testing. We cover some of them below.

Accelerate time-to-market

Today, software product teams commonly do hourly builds and ship customer releases on a weekly or daily cycle. Also, the duration of a user story from conception to customer delivery is now 2-5 days for many software-as-a-service companies. And many of these organizations directly attribute these very short cycles to some manifestation of continuous testing improvements. CT enables high-quality, high-frequency delivery of value-adding user features in each software release.

Building better products

Historically, a conventional product team might work on many low-use features but wouldn’t discover this until the next major release—a waste of too much effort. Today, using CI/CD/CT enables frequent product releases, which results in much quicker user feedback, which in turn drives better feature correction and reprioritization.

Improved productivity and efficiency

In CT environments, testing productivity is significantly better. Only a few years ago, testers might spend more than 20% of their overall effort configuring and fixing test environments. Today, with a good CI/CD/CT pipeline and good tools, many environments can be automatically configured. Also, ops engineers would historically take several days to perform a production release. Now, it can be done with a click of a button.

More reliable releases and lower risk

With CT, the risks of performing a product release are considerably lower and the release process is more reliable. Typically, the deployment process and scripts are tested over many iterations before moving to a production release. Most defects and anomalies will have already been discovered by that point. More frequent releases usually means that the number of code changes is smaller in each release, which also means that it’s easier to find and fix problems.

Improved Product Quality

As we at Functionize are always keen to emphasize, there remains much opportunity in the industry for testing improvements and innovations. But we also acknowledge that overall product quality in a proper CT pipeline can be quite high. For seasoned teams that are disciplined on CT, the number of open bugs for a software that has been released is more than 90 percent lower. In a CT environment—immediately after a code commit—the entire code base is run through a series of various types of tests. If a test reveals a problem, the developers know they must fix it before moving to the next task. This eliminates many bugs that would remain in a conventional release process.

Higher customer satisfaction

With end-to-end CT that truly includes the customers, there is no distrust or tension between the user community and development team because of quality and release disappointments.

Mind the Gap: Achieve better quality software with continuous testing

We encounter it all the time. Too many people still think of testing as a separate phase in the software development process—something that is done after writing the code. “They” code, and then you test. But what we all really want is a continuous stream of defect-free software. If you are serious about that—we most certainly are!—then we humbly suggest that you no longer think of testing as a phase or a separate activity. So, you ask, how can I take a team from conventional testing to CT?

The best way to minimize defects across a product development pipeline (or large system) is to ensure that each segment in the pipeline is free of defects. This is the essence of continuous testing. We must come to view testing not as a separate phase or activity—done by a separate group of testers. It must rather become integral to all efforts through the entire team. Think about it: if you’ve got all kinds of development activity happening every hour of every day of your project, and you seek to improve the quality of those activities, then it is plain to see that testing should not be a “phase”. Testing and quality assurance becomes part of what everyone does, all the time.  

Continuous testing is a framework for running automated tests—as early as practicable and across the product delivery pipeline—in which the results of these tests quickly provide risk exposure feedback on a specific software release candidate. The promise of continuous testing is faster delivery of higher quality software. Functionize can help you move faster toward a continuous testing framework that will progressively deliver better products to your customers.  

Resources: 

https://devops.com/continuous-testing-vs-test-automation/ 

https://www.infoq.com/articles/cd-benefits-challenges 

https://www.infoq.com/news/2015/04/benefits-continuous-testing 

https://dzone.com/articles/continuous-testing-best-practices-amp-challenges 

https://devops.com/continuous-testing-vs-test-automation/