The Importance of Planning your Tests

In many companies, especially smaller start-ups, testing is often an afterthought, with CEOs and managers sometimes viewing it as wasting valuable resources. But as we all know, without proper testing you risk releasing buggy software that damages your reputation and loses you customers. However, there is no doubt that testing can consume a lot of resources and so it’s essential to plan it carefully.

The need for test planning

Testing should be a key part of any project plan, whatever development model you are using. The traditional model for software development as encapsulated in the Kanban board views testing as a distinct phase that happens after development and before some form of monolithic release. By contrast, Continuous Integration and Delivery (CI/CD) releases new code constantly, with testing happening continually during development. In both cases it’s essential to plan the testing carefully.

A well-planned test suite can save significant heartache for companies. However, test planning requires as much care as project planning. Testing should be viewed holistically to ensure all tests are properly thought through. Often individual tests can be combined into more efficient flows that test several functions at the same time. Other times it may be possible to test functionality simply by comparing system state before and after the test.  

UI testing is one area where planning is harder. In a UI test you are not just interested in whether the system ended in the correct state. The process of getting there matters too. Let’s take a simple example. Imagine you have a small e-commerce site you are testing. The test is simply to see if you can order an item and pay for it. So, your test consists of searching for the item, clicking to add it to the basket and then going through the payment flow. The test would seem to be successful if the correct item ends up being purchased. But as this is a UI test you also need to check for some other elements. What if the item is correct, but the photo is wrong? Or the description is in the wrong language? What if there’s a button missing on the page, or in the wrong place? Unless you specifically add checks for these to the test, your test isn’t thorough enough.

Types of tests

As we have seen in previous blogs, there are many types of test to consider when planning. These include:

• Unit Tests which are used by developers to test the functionality of specific software units.
• Integration Tests are used to test that different software modules are working correctly together.
• Functional Testing looks at the functionality of the whole system. These tests should be the central part of your test plan.
• Smoke Tests are used to quickly check that new builds aren’t broken and should be designed to test all essential functionality for the system.
• UI Testing is essential for any software with a major UI element. As mentioned above, it differs from other forms of testing in that it matters what state the system (UI) is in during the tests.
• Performance Testing is critical to ensure your system doesn’t fail the first time it sees a surge in customers.
• Regression Testing ensures new code isn’t triggering old bugs that were previously fixed. It also highlights cases where bugs only become apparent after new code has been added.
• User Acceptance Testing is used to check that your software actually delivers the necessary functionality for the end users.

A robust test plan should encompass elements of all of these. Of course, if your system is sufficiently simple you may be able to combine or collapse some of these, but you should be aware that you are doing this.

Test planning vs. planning tests

It’s important to highlight the subtle difference between test planning and planning your individual tests. Your overall test strategy needs to be planned to make sure you are testing every element that you need to. However, every test you create also needs its own detailed test plan. It’s easy to confuse these two concepts, especially since “Test Plan” is often used interchangeably for both. One way to avoid this confusion is to adopt the phrase “Test Suite” to refer to a set of tests and reserve “Test Plan” to refer to a single test.

A good way to think of this is to imagine each individual test as a piece of a jigsaw. As with a jigsaw, the detailed picture on the piece matters and, importantly, it has to fit exactly into the other pieces round it. In this analogy, the overall test plan is the completed jigsaw, with all the pieces neatly fitting together. However, unlike normal jigsaws, in testing the pieces can be assembled in lots of ways to construct the different tests mentioned above.

UI test planning

As we have already seen, planning UI tests needs particular care. When creating a functional test, the desired outcome of the test, positive or negative, is clear. With UI tests you need to think more carefully. Firstly, you need to know what to expect in the UI. For this, you can use the designer’s wireframes. Next, you need to decide which functionality is most critical, and which elements of the UI need to be checked. Thirdly, you need to note down the series of interactions needed. The important thing is to work out which of these can be left loosely defined (e.g. search for a product) and which need to be more precise (e.g. purchase product X for $20.99 and pay with a Visa card). Finally, you can formalize each set of steps into a test plan.

Traditionally most UI testing was done manually. However, nowadays much UI testing is automated. Selenium IDE makes it easy to record test plans simply using point-and-click. But the problem is that you need to create your test defensively and not make assumptions about the page having loaded correctly. During a manual test, this is easy – it is immediately obvious if the wrong page has loaded, and if you know what to expect it’s also obvious if there are missing UI elements. 

When recording a test with a point-and-click recorder it’s important to build in the same sort of checks. So, after each page change, you should find the elements that you can check to ensure the page loaded correctly. Take the example of the e-commerce site we mentioned at the start. Having searched for a known item, the next step of the test might look like this:

• Check that the correct item appears at the top of the page of results.
• Check that the correct description is shown.
• Confirm that the correct picture is shown.
• Check that the correct price is shown (possibly including a check on the currency).
• Check that the “Buy” button is visible.
• Check that the page header and footer loaded correctly (this can be done by looking for the presence of known elements.).

If you are being really thorough you would include more checks for more UI elements that should be visible in the page.

Other aspects of test planning

There are a couple of other important aspects that go into successful test planning, be it for UI testing, smoke testing or user acceptance testing. Firstly, it is essential to ensure the test environment is as close as possible to the production environment. All too often I have heard of cases where all tests pass in the test environment, only to fail in production due to some missing library, or some library version conflict. A classic gotcha like this was when I was testing a mobile app and discovered that when testing on a mobile network, many API calls were getting blocked by a transparent proxy. This hadn’t shown up earlier while testing on Wi-Fi.

Secondly, you should always plan tests defensively. This means testing for negative events as well as positive ones. Taking our e-commerce example from before, this might mean deliberately searching for products that are missing or are known to be out of stock. Thirdly, having created all your test plans it is important to look over the whole suite as a whole. Often you will then spot ways to save testing time (and resources). For instance, by ensuring that the state the system ends up in after test X is the starting state needed for test Y.

Conclusions

In this blog, we’ve touched on why test planning is so important and have looked in a bit more detail at what makes UI testing particularly challenging to plan. The addition of Artificial Intelligence to UI test creation can help you create more loosely defined tests. But even then, you have to be extremely aware of what you are actually training the system to look for. In conclusion, test planning takes quite a bit of time and effort along with a certain amount of skill. But when done properly it will save you a lot of pain in the longer run.