Beyond the Testing Pyramid – Are Unit Tests Enough?
Automation and machine learning have made shift-left testing easier. But which adds more value, unit testing or end-to-end testing? Read on to find out more.
The software development life cycle (SDLC) has changed since the IT boom of the late 1990s. Back then, everyone used the waterfall methodology to build applications. This phased approach locks down the output of the current phase before moving into the next phase. But nowadays this has been almost completely replaced by agile development. Here, we look at why this affects how you should think about unit tests and test automation.
Looking behind the waterfall
Before we look at the modern use of unit tests and test automation, let’s see why the waterfall methodology was so widely used. Clearly, it had its benefits: its primary goal was quality. Development teams conducted a heavy-duty requirements-gathering and analysis at the start. They had the client sign off on it before they started writing code. They placed an emphasis on testing the code step-by-step. Testing commenced only after completing coding (unit, integration, system, user acceptance). The result was software that reflected the client’s specific requirements accurately. Releases would be an annual, celebrated affair. Multi-year contracts would be set up for maintenance and upgrades down the line.
But during the dot.com bubble, software consumption exploded. The waterfall model became antiquated as it slowed down the pace of innovation. The software developers of today use an agile approach that allows them to release much more frequently.They use continuous integration and continuous deployment (CI/CD) to meet dynamic, ever-evolving user demands. In turn, this means testing has to evolve too.
The unit test: a product of the waterfall model
The unit test is arguably the most valuable discrete process of the waterfall model. Unit tests check every element of the code with the larger goal of driving up quality. For example, testing a button click. Did the button submit? Did it call the event it was designed to? Does the correct text get rendered when you load this page? Does the page load at all? Does the correct image get rendered? Does the URL point to the correct page? Does the title get populated according to the regular expression you have programmed?
The resulting test suite was a composite structure of both manual and automated tests. It would check the functionality of every last feature that the software claims to offer. Coding was complete when unit testing was complete, and all required updates were made.
But how does this work in a CI world?
When you incorporate a continuous integration approach, coding and testing happens hand-in-hand and in shorter bursts. Devs integrate code changes into a shared source code repository at high frequency. This is particularly valuable for SaaS companies who need to release updates as soon as they are ready—daily or even hourly.
The CI approach gave rise to shift-left testing—completing smaller (and broader) tests earlier in the SDLC. Testing gets shifted leftward in the SDLC for continuous feedback to inform design and coding.
Here is where automated testing really adds value.
The role of test automation
Testing automation tools help developers and testers define test cases and script tests. They can create a thorough test plan that will check all corners of the code whenever there is a change. Automation has helped every level of testing – from unit testing to integration and system testing, to regression testing and performance testing, and even user acceptance testing.
The traditional testing pyramid
Traditionally, we follow a pyramid structure when we plan and organize our automated test suites. The majority of automated tests are unit tests—they are the smallest and easiest to build. Smaller pieces of code have fewer interdependencies. They are more resilient and can be scaled up faster.
Successful unit tests take the code through to integration testing, which checks whether units function correctly with each other. Do they work the same way once the code is combined? This is followed by system testing to check the interdependencies between all the parts of the system. For example, querying a database to display user profile data or calling a web service. Each of these tests gets progressively more complex, and so it has traditionally been shown as a pyramid.
Once the test suite has checked all these workflows and interdependencies, the software is ready for user acceptance testing. End users (humans) use the application through the front end, as it's intended, and provide feedback.
The impact of shift left
As development teams adopt a CI approach, where tests run earlier for more frequent feedback, we see a larger emphasis on unit tests. Continuous integration has clear and immediate benefits. But the need to provide continuous feedback during development for quicker releases has resulted in many companies prioritizing unit tests over UI tests. They might do the bare minimum number of UI tests - or skip out on automating UI tests altogether.
Creating UI tests that check workflows across all the application layers takes time. They are also harder to maintain. The pyramid reminds us that tests become more complex as they build up. They become more challenging to automate. UI flows are more fragile and easier to break. It begs the question – are we testing the UI enough?
Do unit tests really drive quality?
The well-intentioned waterfall approach really did drive quality. The accuracy of an application’s functions used to be an indicator of its strength and, back then, its success. But demands have changed, and the scale of testing and delivery has reached new heights. It simply isn’t enough anymore to have accurate functionality. Moreover, today’s applications rely on integrating multiple 3rd party applications and services. Unit tests simply cannot test these properly.
Nowadays, businesses map their strategies to customer journeys. They expect all their touchpoints to deliver a brand-level experience to the customer. A reason to complain is a sure-shot way to tank brand equity. If the user journey breaks at any point, they lose a customer.
Unit tests do not check the integration between all the functional components of the application. They are, in fact, furthest from the end user experience. Today, you need to test the application end-to-end to drive quality. You need to encompass the breadth of UI testing to uncover potential bugs before one breaks the user journey. Insights from E2E testing are most valuable for developers to incorporate into the design.
Integration testing, system testing and acceptance testing all need to form a bigger part of the test plan. Automation plays an important role in making this happen.
Let’s flip the testing pyramid
We are in an era of advanced testing automation. Machine learning technology helps test automation tools internalize and build upon the testing logic that developers create and incorporate it through dynamic application changes.
As we embrace Agile, DevOps and hybrid methodologies for software development, and incorporate continuous integration and continuous delivery, we need to emphasize end-to-end UI tests. We believe the testing pyramid needs to be flipped.
With an AI-powered automation tool, you can get all the benefits of unit tests throughout your testing journey. This means you are more likely to uncover potential issues at different touchpoints along the entire user journey. Functionize has “cracked the code”. Our system is powered by AI, making it autonomous. It dynamically self-heals, so it can recover from most test failures without any human intervention. Tests are really quick to create thanks to Architect, so it’s easy to create the volume of tests a team needs to do shift-left testing right. This opens up test automation to the entire spectrum of business users.
In short, Functionize takes the pain out of E2E UI tests. Our tool helps testers emphasize more UI test cases and find more bugs to fix pre-release. It also makes it easier to maintain tests, making them less likely to break. This results in a holistic, value-generating test suite at a fraction of the cost of a traditional test automation tool.
It’s time to flip the pyramid.
