Achieve Higher Levels of Maturity in Testing Automation

In the age of software development agility, there is more incentive to move toward a continuous delivery model. Continuous Delivery (CD) enables a production-ready software release—at any time. Though it takes time to implement, configure, and remake the team culture, it dovetails well with Agile methodology. When done well, CD significantly reduces the release timeline from weeks to merely a few hours.

“Releasing software is too often an art; it should be an engineering discipline.”  
― David Farley, Continuous Delivery 

Continuous Integration (CI) is the core of Continuous Delivery. CI has a longer history, and it is easier to implement. The objective is to integrate code into a common repository. During the automatic build process, each code check-in is verified to be compatible and functional with the production code base. This verification serves to detect and solve problems much more quickly—at a much lower cost. Many IT and product teams now depend very heavily on CI across their development pipelines. 

Continuous Deployment is an extension of CI, in which code deployment automatically begins the moment it passes all CI tests and verifications. If the verifications are solid, then top-quality releases can move directly to the marketplace as quickly as possible. As you may know from direct experience, CD often doesn’t work out well. This is due to neglect of a vital aspect of software development.

Continuous testing is the missing link

The view of many industry professionals is that software testing has been slow to keep pace with the innovation of Agile development. The problem is not that the testing is inherently faulty, it’s that testing remains an afterthought for so many project managers, developers, and product managers. Most software testing approaches are failing to keep pace with advancements in software development. And yet, paradoxically, the importance of testing continues to increase as software becomes more complex.

If testing is done too far downstream, there is a higher risk that more defects will be discovered and they will be much more costly to fix. Even if developers strive to run manuals tests as soon as practicable, a team that has not implemented continuous testing framework will find that there is significant, time-consuming rework to be done in each cycle. Many such teams still find themselves running tests after each phase—after the test is built, again after the build is complete, and again after refactoring the code. 

Adding to the strain—and time drain—is the fact that more testing is necessary as the software ages and increases in complexity. But, most software companies have limited resources and can’t find the additional time to run more tests. The unappealing options are either (a) to risk some sort of compromise on quality or (b) slip the delivery schedule. 

It is possible—indeed quite feasible—to minimize this dilemma by implementing continuous testing and automating most testing efforts. A continuous testing tool/framework monitors for code changes, then automatically executes tests to ensure immediate identification of any exceptions or issues. 

As software complexity increases and more features are built on top of the older features, more tests must be configured and executed to ensure the same level of quality. Software teams can’t afford to spend more time—since the delivery schedule must remain intact—and many teams can’t add more manpower to design and run all the tests that need to accommodate the feature additions. A typical compromise is to run only a fraction of the tests. Since many of those excluded tests are critically important, the result is lower quality in the release product. Any CD process that is built as such will eventually fail.

Pursuing maturity in test automation

While it is a fact that automation can significantly increase your testing speed and also increase the scope of your code coverage, there is a risk in attempting to automate too many things—or the wrong things—too quickly. It’s also important to avoid having your best testers doing work that doesn’t maximize their talents and skill sets. When you take such risks and fail, a golden opportunity will have been lost. The setback for the team can be quite detrimental.

“Asking experts to do boring and repetitive, and yet technically demanding tasks is the most certain way of ensuring human error that we can think of, short of sleep deprivation, or inebriation.”  

― David Farley, Continuous Delivery 

Whenever possible, any agile software development team should prioritize testing automation until it reaches the point at which it is a key concern for everyone. The extent to which the team works hard in configuring a solid, effective environment will directly determine how many benefits it will reap from testing automation. 

If your team hasn’t put much emphasis on testing in your CD pipeline, then you’re probably don’t have a CD pipeline! There’s no shame in this, lads and lassies. We all crawl before we walk; we can’t run until we master our walk. Below, we explain how to progressively improve your development process with testing automation so that you can enjoy benefits such as these:

• Find issues well upstream, by having developers work with testers to design tests before development is complete.
• Involve QA in the build process, and speed up the release timeline by automating most of the testing.
• Improve code quality by incrementally automating to the point at which the team is testing everything—on every build.
• Minimize risk and gain the confidence that the code is solid on each and every build.

Initial automation steps

At the beginning of your automation effort, there are a number of easy wins that are achievable for the team. Start by having the development team check in their tests and immediately communicate all pass-fail feedback. Failures must be dealt with immediately. When you get the tests to pass, have QA begin building a small set of automated smoke tests.

Initially, it may be easier to execute these tests nightly, but the objective should be to have these running as an integral part of the build process—then automated to run on each successful build. After you reach the point at which these tests pass consistently, the team can incrementally augment the automation suite with more smoke tests. Early on, you can steadily gain confidence by simply adding one small test at a time to the automation suite. Cumulatively, this establishes the foundation of your automation practices.

Gaining confidence in testing automation

When you’ve laid the foundation, you can have QA turn their attention to the various layers of the application. If you are testing an application that will permit the team to separate the integrated and UI tests, you could make the move to distribute these tests and run them in parallel.

“In software, when something is painful, the way to reduce the pain is to do it more frequently, not less.”  

― David Farley, Continuous Delivery 

If you’re like most teams, then your regression suite is a good place to begin configuring to run multiple layers in parallel in the cloud. This will reduce the time to receive results and feedback. Much like automating smoke tests, you can incrementally increase the degree of automation across your regression suite—one at a time if necessary—to reach the optimum level of test coverage. As you automate more regression tests, the team can reclaim much of that saved time for exploratory testing and expand coverage out to the edge cases where you are sure to find the truly the interesting bugs!

Level up to quality gates

A team reaches the expert level of testing automation when it moves to implement and automate quality gates, which enable a team to automatically reject any builds prior to advanced testing or move to the release build. The goal here is to quickly provide early feedback so that any necessary root cause analysis can be done as soon as possible.

At this level, the goal is to enhance the configuration such that only just enough testing is done at each stage before you fan out and parallelize your testing in the next stage. It’s critically important to achieving a high level of confidence at each quality gate—before advancing to the time-consuming, costly automation that awaits further downstream.

Leverage machine learning to improve your automation suite

The ultimate in testing automation is to bring the power of machine learning (ML) into this domain. With highly innovative solutions such as Functionize, the time to leverage ML is now. It is truly feasible to leverage ML to create practical solutions to overcome many of the testing challenges summarized here.

Today, it’s really possible to employ machine intelligence to enable both testers and developers to create reliable, repeatable, automatic tests—in seconds. Functionize will automatically write tests for you, run them, verify them, and guide you to infer actionable insights if something goes wrong. 

There’s no need to worry since there is no danger that machine learning will obviate the need for conventional software testing. What is more likely is that testing will become considerably more challenging as complex applications are tested automatically by machines. The central challenge will be the difficulty in redesigning or containing application functionality to mitigate the undesirable results that arise from many of the cases that an ML engine runs. Many of those ML test cases will be found to well exceed human ability—and thinkability. 

Many testing professionals now approach their discipline deterministically. A quality assurance test will only produce results that a test designer predetermines to be correct— or not. Machine learning overturns all of that since it performs a far deeper examination and analysis. Testing teams will then have to wrestle with a large number of indefinite ML results. This will require a hard rethink about different solutions to the new challenges. Though it brings fresh challenges, there is great value in leveraging machine learning to automate testing to a previously inconceivable level. Are you ready to level up?