Why Test Automation is Essential for CI/CD

Ask virtually any DevOps expert what is the essence of a Continuous Integration / Continuous Delivery (CI/CD) pipeline, and they are likely to focus on integration frameworks, code repository management, and CI servers. Unlikely to be on that list is any mention of test automation, which is a bit puzzling—given the steady increase in software complexity and the pressure to release more frequently.

In this article, we look at how the importance of test automation for CI/CD pipelines—why it is essential, how much should be automated, and what types of tests should be automated. 

Test automation is often neglected

Testing isn’t an exciting topic in most CI/CD discussions, and this is because the primary focus of DevOps is constructing and maintaining the integration-delivery pipeline. There’s no point in being resentful over the reality that—in many cases—DevOps simply neglects the roles of QA and testing. Like many roles outside QA, many DevOps engineers continue to maintain the assumption that conventional exploratory, ad-hoc, and manual testing is quite sufficient.

Why test automation is essential

If your team is developing complex software and looking to ramp up on CI/CD, then it’s quite impracticable to move forward without implementing test automation to a significant extent.

Here are some of the reasons that test automation is essential to a successful CI/CD pipeline. Test automation enables:

• Easy handling of small changes — Even after automating, it will remain sensible to infrequently perform some manual testing—such as usability testing—when an application undergoes a major update. In a CI/CD pipeline, most of the updates and code integrations are small. Test automation alleviates much of the manual testing tedium that comes with small changes and enables the team to continually deliver a steady stream of small changes that have been thoroughly tested.
• Faster tests — Continuous delivery requires speed and minimal delays. Automation produces results much more quickly than manual testing. Parallel automation also shortens the overall testing duration.
• More consistency — An effective CI/CD pipeline must have few variations or anomalies. Working hard to maintain process consistency ensures that expectations are met all the way downstream to the release.
• More agility — In a robust pipeline, it should be practicable to adjust frameworks, configurations, and tools as new technology becomes available or requirements change. Slogging it out with so much manual testing makes it very difficult to be agile. It is terribly inefficient when the team has to hold back to reconfigure or rewrite test suites at times that the CI/CD toolchain needs an update. With a significant degree of test automation in place, most of the reconfiguration can be done automatically.

How much can a team automate?

The best approach entails knowing what types of tests to automate and which types should remain manual or exploratory. Elfriede Dustin, in his book Implementing Automated Software Testing, writes that:

“Not all the tests for a project can or should be automated. The best candidates for AST [Automated Software Testing] are those that are repeated the most often and are the most labor-intensive. Tests that are run only one time or infrequently will not be high-payoff tests to automate, unless the given context warrants for automation, for example in the case of the test being difficult or cost-time-prohibitive to run manually. Further, test cases that change with each delivery are also not likely to be high-payoff to automate if the AST needs to change each time.  

“Our experience has been that 40% to 60% of the tests for most projects can and should be automated. An initial top-level assessment should be developed for each project. As you gain experience in implementing AST for more projects, you will continue to refine your ability to accurately gauge the degree to which you can apply AST to the test program, based on your own historical data.”

What is test automation?

If you’ve got this far into the article, you may appreciate some clarity on our beloved topic of test automation. What, exactly is it? Generally, it’s the opposite of manual testing, which is done by a human tester on a computer or mobile device. The basic method of manual testing is to attempt various use cases and input variations, watching for functional failure or anomaly—and then comparing the results to the requirements. Test automation, by contrast, is the use of scripts, tools, and frameworks to execute preconfigured tests against an operating system app, web app, or mobile app. The automation framework captures all results and presents detail reports for review by human testers.

Of course, it’s very necessary to invest adequate time testing an application to reach a high level of quality. While that is the primary goal, it is now becoming equally important—in this era of increasing software and systems complexity—to reduce manual testing effort and automate as much as possible. As a team makes its initial foray into the realm of automation, a QA team will encounter a number of challenges—which we have written about elsewhere in this blog. It’s here that many lose confidence and think about giving up on test automation. 

We can tell you, from our own experience here at Functionize and that of our customers, that persistence through the initial challenges can definitely reap a huge payback. Over time, the effectiveness of manual testing reduces, while that of automated testing only increases.

A solid approach to test automation

The best technical approach entails knowing which tests the team should automate and which should remain as manual tests. Tasks that are repeated most often and are the most labor-intensive are often the best candidates for automation. Let’s have a look at various types of tests in the context of automation.

Unit and component testing

Unit tests verify that individual units of source code are functionally sound and ready for integration into the repository. Typically, these are elemental tests that focus tightly on locations where bugs might be lurking—far upstream in the delivery pipeline. Quick feedback is vital here, so unit tests should be extremely fast. Collectively, unit tests should cover about 80% of the codebase—which gives solid confidence that the application is likely to function properly further downstream. If your team doesn’t automate unit tests, the chances of enduring success are very slight.

GUI Testing

GUI testing ensures that the frontend interfaces of an application will appear and operate according to product requirements—in all target browsers and platforms. For those who still do this manually, this involves the tedious task of rendering all pages of the app in various combinations of browsers and platforms. Virtual machines have made it possible for teams to test multiple combinations in parallel, at high speed. This is a core strength of the Functionize platform, as we exhibit in our recent article.

API testing

API testing validates the interoperability and logic layers between software components and systems. Rather than simulating keyboard or mouse input, API testing simulates calls to API resources (often a special URL) and verifies the response to those calls. A potentially serious challenge to effective API testing is proper test environment setup—which must be done with much more programmatic configuration and much less visual interaction. This can be one of the more difficult aspects of test automation.

Functional Testing

In functional testing, the multiple and various functions of the app are compared stringently with the requirements or specification documents. A tester stands in the role of a user or customer, who interacts with the features of the application, inputs are given, and the output is compared with the output specifications. Functional testing avoids any consideration of the application internals, but rather focuses on the user functionality. For a web app, an example of a functional test is that a tester manually navigates throughout the application to verify correct operation. However, since test automation is the best way to ensure frequent testing, it is best to automate functional tests whenever practicable.

Regression Testing

The aim of regression testing is to reveal bugs that might have crept in following changes to the app—such as enhancements and previous bug fixes. Prior to a new release, existing test cases should be run against the release candidate to validate that all functionality in the previous release continues to function correctly. As release frequency increases, a team may shortcut on regression testing to save time. This typically allows bugs to pass through to reach the end-users. This is another type of testing that is very suitable for automation. Functionize employs a unique Adaptive Event Analysis engine that employs machine learning to search deeply into applications and automatically manage necessary changes to regression tests.

Can you afford to delay test automation any further?

Software development teams that realize the need to transition to CI/CD can start small—with some of the regression tests, for example. There will be some challenges, but these will be growth cramps that are to be expected as your team strengthens its testing suites with automation. The benefits will eventually become evident as your investment in the right framework and the best technical approach for your environment.

While Selenium and Appium are respectable solutions for testing web and mobile apps, they are unquestionably tedious to maintain. Functionize's ML Engine gives you more power and flexibility and helps you greatly minimize the burden of manual reconfiguration and maintenance. Companies that invest substantially in test automation initiatives will attain a competitive position that fully exploits their CI/CD pipelines. This will translate directly into faster delivery of higher quality products and delighted customers.