Data-Driven Testing: Why it's not Overkill for Your App

Modern web applications are getting more and more complex. They need localization to ensure users get a good experience wherever they live. They also need to cope with rich data and interactive elements. This poses increasing challenges for test automation. For one thing, you need to repeat the same tests many times with different data. For another, you need to handle large amounts of test data across all your tests. You can solve both of these using data-driven testing.

What is data-driven testing?

Data-driven testing (DDT) splits the test script logic from the actual test data. The data is stored in a database along with the expected outcomes of the tests. The script simply reads in the required data and outcomes when it runs. This simplifies test script coding, while enabling thorough testing with realistic data.

There are several reasons to do this:

• If developers use hard-coded sample data for testing, the application could break.
• Hand coding test data takes a long time and is prone to errors.
• DDT makes it easier to create test data that mimics real application data.
• You can include the test data and verification data in one file.
• Developers can drastically reduce the amount of coding they need to do simply by separating the test code from the test data.

Why do we need data-driven testing?

Test automation often requires test data to validate that your application is working. For instance, your application may require a user account to access certain functionality. Traditionally, this test data is generated when you create your test script. However, this is pretty limiting. Firstly, as your application evolves, you may need the data to evolve with it. As an example, you might be testing an eCommerce site. If your test relies on adding an item to the cart and that item no longer exists, the test breaks. Secondly, often you need to test one scenario many times with different data. A perfect example here is testing localization aspects such as language or currency choices. This is where data-driven testing comes to the fore.

How does data-driven testing simplify test script coding?

Test engineers may struggle to understand how DDT can help simplify test creation.  

The aim of DDT is to simplify the test logic by separating the input and output data from the actual test. However, this can seem like overkill at first. The simple example below helps illustrate the concept, but DDT really comes into its own with complex scenarios, such as localization testing. 

Some people may think that data-driven testing is overkill because it’s too time consuming to set up. Or, they may think their application isn’t complex enough for data-driven testing. But data-driven testing is actually very easy to set up and the benefits are worth it. 

Let’s look at a simple test for a user login. The test data is simply a username and password, so the logic table would look like this:

UsernamePasswordResultOKOKPassOKInvalidFailInvalidOKFailInvalidInvalidFail

Coding this is simple. You start by creating an Excel spreadsheet listing the four test cases above. Of course, you need to use appropriate username/password combinations. The results of each test can even be appended to the spreadsheet as the test loops through the testing scripts.  

The test script instructions for this test would look something like:

• Launch application

• Input content of cell A:(01+i)“Username” from spreadsheet

• Click “Tab” button

• Input content of cell B:(01+i) “Password” from spreadsheet

• Click “Tab” x2 so “OK” is highlighted

• Click “Enter”

• Check result & append to source spreadsheet at xx:xx and add

• Goto next loop

This relatively simple string of input logic can be reused again and again with little tweaking. Moreover, it can be adapted to test similar input from (for example) another part of the world, or speakers of a different language. Ultimately, you are simplifying the test development and deployment process. This results in a functional framework which offers the maximum flexibility for testing.  

DDT becomes a powerful and convenient tool because you are able to think about the test logic and script separately. This allows test engineers to orchestrate tests simply by changing the test data. This is essential if you are going to scale up to running thousands of tests in parallel. This also makes it an essential part of modern AI-driven test solutions, such as Functionize.

Common misconceptions about data-driven testing

Lots of test engineers think that data-driven testing...

1. ...takes too much time and effort to set up and deploy
2. ...is hard to learn
3. ...needs setting up separately for each new project
4. ...puts customer data at risk
5. ...isn’t that different to any other way to set up a testing framework

Fortunately, all these are misconceptions when it comes to test scripting. Moreover, as we will see later, DDT is just as applicable to modern AI-driven approaches, such as Functionize. 

DDT is hard to set up. Data-driven testing doesn’t take any longer to set up and deploy than other test approaches. Effectively, you are trading off the time taken to set up the test data with time needed to write and update the tests. In the long-run, DDT wins because it allows you to update the test data without needing to modify the test scripts. 

DDT is hard to learn. The theory behind data-driven testing is relatively simple. In fact, writing tests for a data-driven testing framework is pretty easy, and guiding an AI platform like Functionize to write them is easier yet. Not only can you build complex test cases using simple boolean logic, but you can do all of this with the help of Functionize’s powerful AI.  

Data-driven testing is actually really easy to learn. The alternative is to hand-code the test data into the testing scripts one line at a time. Something no sane person would welcome in their working life.  

DDT needs to be set up for each team and project. One of the advantages with data-driven testing is the way it simplifies sharing of test data between teams. With no specific reason to keep data and test scripts siloed, test scripts and databases can both be re-used wherever they’re called for. Obviously, it’s quicker to create one shared database, rather than creating individual databases for each team that needs it. This also reduces the risk that different teams are basing their testing on different data. 

DDT puts customer data at risk. DDT is no riskier than any other approach to testing using realistic data. Data protection laws mean you should never be using real-world data for your testing anyway. Instead, you should generate good-quality, realistic test data that is representative of your users. This is actually far easier to do with DDT. 

DDT isn’t that different from other approaches. It can be easy to dismiss data-driven testing as a gimmick. However, DDT is actually a completely different paradigm and requires a different way of thinking. By focusing on the data you are forced to focus on the expected results. This means you are creating better tests that are also easier to maintain.

Functionize: your choice for data-driven testing

One of the core tenets of data-driven testing is that the test logic is always managed separately to the test data. Doing this makes your life much easier when you are trying to manage testing across large, complex software projects.

Functionize helps you reduce the time and effort spent setting up and running tests using orchestrations. Test data management (TDM) is an integral part of Functionize, and allows test engineers to get the most out of its advanced DDT features. Setting up and using your test data consists of three main stages: 

• Data source creation: If you don’t have the right data source already on the system, you will need to upload your new information. With Functionize, this will be a .csv, .xml or, for more complex data, a MySQL database. Creating synthetic test data can be quite challenging but also quite fun. There are tools that can help with this. Once created, it's easy to add a new data source to Functionize with only a few clicks.

• Generating random data: Often, your test will need to use random data, such as passwords. Functionize includes inbuilt random data generators that can create lots of different types of data. These include random strings, functioning email addresses, cell numbers for SMS verification (2FA), and time/date stamps. This makes it easy to dynamically generate any test data you need.
• Using test variables: One unique feature is the ability to use previous values later in a test or in subsequent tests. This includes comparing against the result of a previous test step and storing named test variables. This allows you to leverage DDT even without external data sources. For instance, you might create a login with a random name and email. These are then stored as variables and reused to verify the user profile screen is correct.

• Test data mapping: Test data mapping is an essential part of creating data for use in your test scenarios or orchestrations. Overall, there are two types of data mapping you will come across: read access and write access. Read access deals with read-only data strings, e.g. inputting an email address. Write access temporarily stores data that can be reused, e.g. storing a random password that was used to create an account and then reusing that password to log in.

Additionally, there is both automated and manual mapping available. Which you should choose is very much up to you, and will depend on the rest of your testing scenarios or orchestrations. To get the most out of TDM you will need to understand both.

• Orchestrations with TDM: DDT involves more than merely importing a database. A key aim of DDT is trying to cover as many scenarios as possible in one orchestration. This requires you to think carefully. Fortunately, orchestrations are easy to create in Functionize. The advanced TDM options in Functionizre make it ideal for data driven testing at any scale.

Now you have some knowledge about data-driven testing and how to implement it, you will be able to get even more out of Functionize. If you want to learn more and see this in action, simply book a demo with us now.