Testing—the vaccine to reduce risks for your software releases

Recently, the concept of risk has been in the news. How can someone weigh up the risk of dying from COVID-19 against the risk of dying from the vaccination? Here, we look at why understanding risk is so hard and how it relates to software testing.

Risk is a fact of life. We are constantly surrounded by risks. Even a simple action like putting on your pants carries a risk. Every time you get into a car, you have a finite risk of death or serious injury. But we humans are remarkably bad at assessing these risks. We lack the mental tools to properly compare one risk against another. Indeed, this lack of understanding of risk has led to the University of Cambridge appointing a Professor for the Public Understanding of Risk.  

The current holder of that chair, Professor David Spiegelhalter, has spent the past year trying to educate people about the risks posed by COVID-19. He is fighting a battle against the vast tide of misinformation, rumor, and downright deception out there. For instance, debunking the myth that it is “only as dangerous as seasonal flu” (in fact, it’s actually a good order of magnitude worse). More recently, the target has become misinformation and excessive caution surrounding vaccines. This has been particularly a problem in Europe, where the AstraZeneca/Oxford vaccine has been banned in some countries due to a small risk that people may die from blood clots triggered by the vaccine. While deaths from the vaccine may seem terrible (and avoidable), the actual risk is over 100x smaller than that posed by the virus.

Understanding risks

One good way to measure risk is the micromort. 1 micromort equates to a 1 in a million risk of dying. For COVID-19, the risk is strongly correlated with age (although there is evidence that some new variants are more deadly for younger people). While accurate numbers are hard to find, it seems that COVID-19 varies from 30 micromorts for children up to 54,000 micromorts for seniors. By contrast, the risk of dying from blood clots following vaccination seems to be under 10 micromorts.

The following chart (from here) gives us another way to interpret this risk. It compares the reduction in risk of ending up in intensive care with COVID-19 against the risk of getting a blood clot from vaccination. In this case, the assumption is that there is a medium incidence of COVID-19. 

As you can see, over the course of a year, even young adults will see a much greater benefit from being vaccinated than their risk from blood clots. And that benefit rapidly increases with age.

Risks and software testing

By now you might well be wondering what all this has to do with software testing? Well, software testing is rather like vaccination. It’s something you do to protect yourself against some future risk. In the case of testing, you are protecting yourself against software bugs. In the case of vaccinations, you are protecting yourself against biological bugs. And just like with vaccination, it can be hard to understand the benefits of testing.

So, what is the software testing equivalent of the micromort? What is the easy way to quantify your risks of getting it wrong? Well, one good candidate might be the number of bugs found by your testing. Each bug you find is one fewer for your customers to find. Another measure is the amount of your software that is actually being tested, known as coverage. The problem there is that modern software is so complex it is actually impossible to test every possible user interaction.

Mitigating risks by testing

The one thing we can state with confidence is that testing is fundamental for all software. Moreover, the better you can automate it, the more effective the testing will be. If we stretch our vaccination analogy further, software testing is like vaccinating against seasonal flu. The flu virus is capable of mutating extraordinarily fast. Therefore, every year you need a new vaccination. In the same way, each time you release new features, your software has changed. In effect, you have created an opportunity for new bugs to appear and old bugs to reappear. That’s why you need to make sure you do thorough regression and progression testing.

How do you prove the benefits of testing

Properly demonstrating the benefits of testing can be really hard. Often, QA managers find themselves having to fight for resources, or being pressured to release software before it has been properly tested. So, how can you prove that testing was worth it? After all, you are in something of a catch-22. If you find lots of bugs, you are the bad guys for delaying the release. But if you don’t find bugs, the heroes are the developers. And if a bug is subsequently found in production, you are back to being the bad guys for not finding it!

We explain more in our eBook on How to demonstrate the value of your automated testing. In a nutshell, the best thing to do is to show how your testing efficiency impacts your business. In turn, that means maximizing the use of automated testing, ideally using a smart testing tool like Functionize.

Conclusions

Risk is one of the hardest concepts for people to grasp. It is purely abstract and comparing or quantifying risks is hard. That has been shown time and again during the COVID-19 pandemic. But as we have seen, it also applies to software testing. Hopefully, this blog has given you a new way to look at the link between testing and risk mitigation.