Smoke Testing Suite: What it is, Why You Need it, and How to Automate

This article can serve as a simple guide you can read and share with your team about the advantages and best practices for reliable smoke testing. We also have a free offer at the end to a limited number of qualified teams each month, so be sure to read all the way through and register for the free offering.

What Are Smoke Tests?

Smoke tests are tests that development teams create in order to quickly verify that basic functionality in a software application runs properly or, if the tests fail, reject the deployment until the basic errors are addressed.

High-quality smoke tests share some common characteristics and areas of coverage:

1. Clicking or navigating through many or all screens of an application
2. Testing that key functionality, such as signup forms, adding to shopping carts, exporting files, etc. work
3. Verifying correct layout and visual correctness

Additional, high-level functionality to include in your test suite will depend on your application and the needs of your users

What is a Smoke Test Suite?

A smoke test suite combines a number of smoke tests into a small suite of tests so that when the application is ready to deliver to staging or pre-production environments, developers will know in a matter of minutes whether the latest version of their application is ready for further QA and review. Smoke tests are also used at times in emergencies or when teams want to release a hot patch to their production applications. This last practice is not recommended and there are better ways, outlined in the last section of this article under "How to Get Started with Smoke Testing."

Benefits

There are many benefits to smoke testing but the most prevalent include:

1. Happier teamsTeams that know the quality of their build before they start the actual QA process are much more productive and happy.
2. More effective QA as a wholeIf your QA process is wasting cycles by attempting to run a larger test suite when there are issues that could have easily been caught in a smoke suite, then your team is going to be crippled with quality problems until you put into practice a reliable smoke testing practice.
3. Earlier detection of bugsSome teams among our customer base report that they are able to uncover and fix as much as 80% of the bugs that they encounter overall simply by dedicating the proper amount of time to performing smoke testing. As is true in many areas of business, this resembles the Pareto principle of 80/20. In this, smoke tests can be viewed as the 20% (likely a lower percentage in reality) of the total test cases catching 80% of the bugs in your application. That is a healthy investment to make.
4. Faster troubleshooting of new and regression bugsFor those bugs that do occur, teams can troubleshoot and perform root cause analysis far more quickly after having learned from the results of their smoke suite than they would have had they simply tried to run their entire testing suite. This is due to the high coverage, shallow depth nature of smoke testing suites. Think of this test suite as a map or lay of the land in your application's quality.

Manual Tests

A common first step for companies who want to build a suite of smoke tests is to first start with manual QA. This is relatively easy to start with and, particularly in younger companies, people who aren't engineers or QA testers will frequently get pulled into performing smoke tests.

As teams mature, they may choose to hire a QA manager or a manual tester. This is a good first step, even you plan to automate in the future because we will show you how to turn those manual QA testers into automation engineers as quickly as possible. There are two major benefits to starting with a manual process:

1. Non-engineers are able to own the testing process, freeing up engineers to build product
2. The people who define manual smoke tests are the best people to create automated tests, provided they have the right tools

Blog post

Smoke vs. Regression Tests

Read now

What to Include?

The main point of a smoke test suite is to let teams know whether or not they should continue testing their entire application. It's also important to understand what level of testing should be included.

There are many types of QA testing but to keep things simple, let's look at Unit Testing vs. Functional UI testing. 

Unit Testing is a best practice among agile software development teams and is something that we won't cover in great detail in this article.  Unit tests are automated tests that happen at the source code level in a project and should be executed even before the smoke suite is run. 

The best fit for smoke testing is at the Functional UI layer. Functional testing is all about testing the user interface (UI) in order to uncover new bugs (exploratory functional testing) or ensure that previously known bugs have not found their way back into the application (functional regression testing).

What is the Right Number of Tests to Include?

You want a smoke suite to be an easily manageable number of tests that can be executed quickly, regardless of whether the smoke suite has been automated. The recommended number of test cases is generally 20 on the low end to around 50 on the high end. Any less than this and you're probably not getting enough relevant coverage, and any more than this and you're probably trying to cover too much ground at this stage.

Automation

Once you have your suite of smoke tests defined, you're going to want to think about automation. Given that the point of smoke testing is to determine quickly the state of the release and its readiness for full QA, you don't want your suite to take more than a few minutes to execute. If you're running 50 tests, and it takes half a day (or more) to run these tests, then your testing cycle is going to be too slow. This is why you want to automate.

Easier said than done for many teams. The process of automation typically involves yet another software engineering project. The most popular tools for building new testing automation focus on browser automation. The most popular browser automation framework today is Selenium, which is an open source project that enables developers to automate browsers. This is important because, in order to properly test modern web applications today, you have to emulate user behavior. The tricky part about this is that when users interact with an application in the browser, there are lots of hidden things happening, such as JavaScript events firing, asynchronous calls being made to APIs, real-time font-renderings, CSS-based animations and much, much more. A proper automation framework will enable teams to both emulate user behavior and get visibility into all of these non-visible phenomena.

Common Challenges and Solutions in Automation

There are many challenges with Selenium and Selenium-based automation projects. As applications continue to increase in complexity, these problems will multiply. Selenium itself is buggy across browsers, and your typical Selenium-based automation projects are brittle, for various reasons. We're solving these and many other problems related to testing automation, so talk to us if you are trying to improve your automation.

Whatever automation framework you end up choosing, automating both your smoke suite and your entire testing suite after that should be top priorities for your team. 

There are many good reasons for automating your smoke testing but the best reason is that you can get through tests faster. You should aim to complete this set of tests in 5 minutes or less.

Opportunities to Improve

Once you have your suite created and an automation framework selected, you'll want to think about a couple of key areas in order to keep your tests running healthy and providing valuable feedback:

1. Test creation and maintenance
2. Continuous Integration tools such as Jenkins and CircleCI
3. Daily execution
4. Create a penalty on developers whose contributions don't pass

Getting Started

The most important thing to remember is that speed is better than thoroughness when it comes to smoke testing. Don't worry about trying to get everything 100% right the first time, you can always improve later. Just think about the things you would normally do in your application and make a list, where each line is a separate test. For example, if you were writing a smoke test for Shopify, the popular e-commerce platform, you might write your initial test plan as follows:

1. Go to home page
2. Register for an account
3. Activate your new account
4. Log in to your account
5. Create a new store
6. Add one item to your store
7. Publish your store
8. Go to new store, add an item
9. Go to shopping cart and start checkout process
10. Test credit card and final payment process through checkout

As you can see, these are very simple sets of tests that you can easily pass to a manual QA test to perform, at a minimum, without much instruction. Even if you were going to go to the trouble of writing Selenium code, you could handle these steps easily. There is quite a bit of mess, however, to deal with in setting up your browser testing environment, dealing with a new code repository, and more.

With Functionize, it would take about 30-60 minutes to completely automate this smoke suite, which is something we will cover in our offer below.

Recommended Next Steps

Once you have your smoke suite created manually and then have automated it, think about a few next steps. A great next step is to integrate your suite of tests into a continuous integration (CI) tool. CI tools will kick off the test, and send you reports and alerts about the outcome of the test.

Regardless of whether you use CI, you should run your smoke test, at a minimum, on every new build. Ideally, you should run it daily, both in pre-production and production environments. Why run a QA suite in production environments? We call this process deep monitoring. There are many things that can go wrong in a production environment. CDNs can go down. Font servers can go missing. Hosted JavaScript files can disappear. External APIs can fail. You get the picture. A smoke suite with proper alerts in production will give you powerful insights to the UX of your application. High-quality UX, quite literally, is money in the bank.

Our Offer to You

Before we started Functionize, the best answer would have been to build an automation project using a framework like Selenium. We've made that requirement go away completely. With Functionize, we can help you create a free smoke test suite of up to 20 tests. You're then free to use it either standalone using our internal CI tools or with your own CI tools. You can use it for both pre-production and production environments, up to once a day.

In order to see if you qualify, just fill out your name and contact information. You'll get an email from us. Tell us you read this post and that you want to qualify for the free smoke suite. 

We'll ask a few simple questions about your team. We only provide this to a limited number of teams per month, so the sooner you apply, the better.  Sign up here!