Top 4 Benefits of Smoke Testing, and How to Automate

Smoke testing is a preliminary level of testing that ensures all basic components of an application are functioning properly. A smoke test does not penetrate into details of the app, but only tests major functions.

The phrase “smoke test” originally came from testing hardware, in which engineers performed quick initial tests to confirm basic, safe functioning of a hardware test unit.

No evidence of smoke means that additional tests can proceed. Today, rocket engineers perform serious high-volume smoke tests. While QA professionals don’t actually expect to see any smoke coming from their software, they do need to know if it’s sensible to move ahead with exhaustive testing.

In this article, we invite you to think more carefully about the practice of smoke testing and how it differs from other types of testing. We’ll also consider the benefits, outline best practices, and look at the importance of smoke testing in any automation effort. 

Smoke testing: Does anything smell funny?

Smoke testing, also known as build acceptance testing (BAT), is a critical aspect of quality assurance that delivers quick and decisive answers on the viability of a particular build. If all the tests in the entire smoke test suite pass, then the build is acceptable for exploratory and regression testing further downstream.

For web and GUI applications, smoke tests will typically cover essentials such as:

• Navigating through many of the foundational pages and clicking on key areas.
• Verifying the correct layout and accuracy of all visual elements.
• Exercising key functionality such as signup and login forms, additions/subtraction to shopping carts, checkout, return to shopping, and file exports.

Any additional, critical functionality that might be part of a smoke test suite depends on the website or application under test, together with user profiles and expected user behavior. More on this below.

Keep in mind that smoke testing is not sanity testing, which a team performs on a software build after bug fixes have been applied to a build and it passes through the smoke test successfully. A sanity test ensures that all non-deferred bugs have been fixed so that the regression suite can be run with confidence.

How important is a smoke test, really?

Of course, thoroughly testing your applications is important for ensuring good quality. Performing a suite of smoke tests is even more important since it is sensible to get an answer to these preliminary questions before moving onto other tests:

• Can the application successfully initialize without error?
• Does the login process function correctly?
• What is the result when you click the primary buttons or menus?

The main reason to run smoke tests is to get rapid feedback—in only a few minutes—that the build is essentially sound. The primary purpose is to get this feedback to the testers immediately so that they don’t potentially waste hours to get the same critical answers.

The benefits of smoke testing

Here we summarize only the main benefits of smoke testing.

1. Detect show-stopping bugs much earlier

Some Functionize customers report that they can uncover and fix as many as 80% of the bugs they discover simply by configuring and executing a solid smoke testing suite. This corresponds well with Pareto principle of 80/20. For many teams, smoke tests might be covering only 20% or less of all test cases and yet catch 80% or more of the bugs. This alone makes it smoke testing efforts worth the time investment.

2. Improve the effectiveness of the QA team

If your QA process is potentially wasting effort and time by only running a larger test suite when many of the issues could be discovered with a smoke test, then you are probably wasting resources and foregoing time savings that you could use to perform other tests or maintain your automation toolset.

Blog post

Smoke vs. Regression Tests

Read Now

3. Faster troubleshooting of new and regression bugs

If any are found in smoke testing, the development team can get to work much earlier on troubleshooting and performing root cause analysis—instead of waiting until after the results of the full test suite. This is due to the high coverage, shallow depth nature of smoke testing suites. Think of this test suite as a sketch-mapping of the quality of the application. If the build is somewhat viable, then more efficiency can be gained if QA continues with (partial) regression testing on that build while developers fix any smoke-test bugs. After fixing those bugs, the developers can then turn to fix any bugs that QA has found in the regression testing.

4. Cultivate a happier, high-productivity QA team

QA teams will be more productive and have higher job satisfaction when they can have more confidence about higher-viability builds that successfully pass through the smoke test suite.

Carefully shaping the scope of smoke tests

It’s important to understand what types of testing should be part of the smoke test suite. For any particular application, there are many types of tests that would be suitable across the delivery pipeline. Let’s compare two major categories: unit testing and functional UI testing.

Typically built and run by the development team, unit testing is an important, frequent, and common task in a development team. Though we won’t go into detail here, it’s important to realize that unit tests are often automated tests that examine source code prior to check-in and prior to execution of the smoke test suite. 

For most graphical user interface applications, smoke testing will include a number of functional UI tests. This is functional testing, not code-level testing. The goal is to put existing and new UI features through a regimen of basic tests that might uncover bugs that have crept into core features. 

We won’t go into detail, but consider that it may also be sensible to include a number of smoke tests that perform elemental validations for any API, logic, or service layers. Database and integration smoke tests may also be necessary.

Sizing and automating a smoke test suite

To be efficient and purpose-built, a smoke test suite should contain an easily manageable number of tests that execute very quickly—even if it has been fully automated. A good range for the number of test would be from about 20 to 50 tests. Too little or too much coverage might defeat the purpose of this preliminary test suite.

Ideally, each smoke test should meet these criteria:

• Tests core features only
• The test should be reproducible indefinitely
• Very fast execution
• Should generate few or no false positives

If you haven’t given it much thought, consider automating your smoke tests. We understand why this may seem to be much easier said than done since many automation efforts are full-blown projects. For non-trivial applications, it’s worth the extra effort. There are many good reasons for automating your smoke testing but the best reason is that tests will run faster, more consistently, and return results automatically. You should aim to complete this set of tests in 5 minutes or less.

There are a number of automation tools available, and a variety of product claims across a wide range of tool vendors. Most well-known tools focus on browser automation, and the most popular browser automation framework today is Selenium. Now, keep in mind that you must mimic user behavior to properly test modern web applications today. What is not intuitively obvious to everyone is that user interaction with a browser application is only a fraction of the events that are occurring. There are many hidden aspects such as JavaScript events firing, asynchronous calls to APIs, real-time font-renderings, CSS-based animations and many more operations. A comprehensive automation framework will enable teams to both emulate user behavior and maintain clear visibility into all of these non-visible elements and operations.

Challenges with conventional testing automation

There are many challenges with Selenium and Selenium-based automation projects. As applications continue to increase in complexity, these problems will multiply. It’s no secret that Selenium itself can be glitchy across various browsers types and versions.  Many Selenium-based automation projects are brittle, for reasons that are too numerous for us to cover here.

Functionize has solved many of the problems in this space, and we are innovating at a very rapid pace to bring the power of machine to testing automation.

Smoke test maintenance

After creating and automating the smoke test suite, it’s important to keep thinking about the health and relevance of those tests by focusing on:

• Test creation and maintenance.
• Daily execution.
• Assess a penalty on any developer whose contributions fail to pass.
• Consider continuous integration made possible with such as Jenkins or CircleCI.

Getting Started

One of the most important guidelines for smoke testing is that speed is better than wide coverage. Don’t attempt perfection in your first attempt, since this will be a continuous improvement effort. As a start, make a list of normal user tasks and ensure that each task is represented by one or more of the smoke tests. For example, if you were crafting a smoke test for a Shopify e-commerce implementation, you might write your initial test plan as follows:

1. Go to home page.
2. Register for an account.
3. Activate the new account.
4. Log in to the account.
5. Create a new store.
6. Add one item to that store.
7. Activate the store.
8. Go to the new store and add an item to the shopping cart.
9. Go to the cart and start the checkout process.
10. Test a payment method and finalize checkout.

This is a good beginning for a smoke test suite. Even when going to go to the trouble of writing Selenium code, this would be fairly easy. The trouble comes in having to configure the browser testing environment, dealing with a new code repository, and other headaches.

By contrast, it would take less than an hour to completely automate a smoke test suite with Functionize.

Next steps

After you got a solid, fast, automated smoke test suite in place, a good evolution would be to integrate your full suite of tests into a continuous integration (CI) tool—which can automatically initiate the test suites, communicate with alerts, and finalize with reports concerning the results of all tests.

At least once-daily smoke test execution is essential, at a minimum, on each new build of each application—in both staging and production environments. Running tests in production are not superfluous. There are many things that can go wrong in a production environment: CDNs go down, font servers go missing, hosted JavaScript files infrequently disappear, external APIs fail. A production smoke test suite configured with proper alerts will provide additional keen insights to the UX of your application. Surely you’ll agree that more UX intelligence is— quite literally—more money in the bank.