The Impact of Cognitive Bias on Software Testing

In many aspects of personal, public, and business life, pursuing objectivity or fairness often means finding a common ground. The ideal is to find a state of neutrality in which everyone can work and communicate in an environment that is free of bias. The problem is that this is simply unachievable because human and organizational bias is pervasive, even within technology disciplines that appear unassailable. Software testing is subject to cognitive bias throughout the phases of test creation, execution, and consumption of results.  

A bias is a particular outlook or inclination of temperament. It’s a natural tendency to view the world through a lens that has been colored by past experience. Each human being has biases, and biases can multiply—or strengthen considerably—in the context of an association, organization, team, or company. It’s very difficult for any of us to modify the biases in ourselves. What is much easier—and more effective for teams that work together—is to acknowledge bias. Knowing that bias exists, efforts can be made to mitigate the detrimental effects that biases have on productivity and quality. This can be quite valuable for software development teams, especially for improving testing efficacy. 

Philosopher of science Karl Popper wrote: 

“If we are uncritical we shall always find what we want: we shall look for, and find, confirmations, and we shall look away from, and not see, whatever might be dangerous to our pet theories.” 

Though we don’t give it much thought, each of us has many of these “pet theories”. One of the few things that can be said without bias is that every person maintains various types of bias. 

Software testing is both art and science, a profession that consists primarily of making choices, assessments, evaluations, and conclusions about software features. This all feeds into what types of tests are built and how they will be run. Although modern development methodologies such as Agile can help a team improve the efficiency of all these tasks, it remains a possibility that there are big holes in test coverage. These holes are usually the result of omissions by QA, but the QA team may be entirely unaware that it is actually their unintentional neglect that is the root of the problem. 

There are many types of cognitive bias that can affect software QA teams, including:

• Confirmation bias
• Congruence bias
• Survivorship bias
• Fundamental attribution error
• Negativity bias

Let’s have a brief look at each type.

Confirmation bias

Confirmation bias favors the selection of tests cases toward those cases that testers predict will execute according to their expectations. This typically occurs in manual testing, if only because—in the minds of the testers—an adequate level of testing is getting done in a modest amount of time. For many teams, this bias also affects the formation of the regression suite (which contains tests with known outcomes). The effect of this bias is that there is no incentive to test for a wider range of unexpected or surprising results.

This bias is perhaps the most difficult to mitigate, because of the tendency toward affirmation, confirmation, and success. The team wants the software to work and is drawn magnetically to the evidence that confirms this desire. A good approach to deal with this urge is to continue reviewing all assumptions and explore other ways to test “outside the box.”

Congruence bias

This is similar to confirmation bias, in which testers plan and execute tests according to their own readily apparent hypotheses and avoid considering the alternatives. Essentially, QA tends to validate functionality in alignment with its expectations. The team doesn’t bother to push further to see if there might be various other ways to test, or possibly other ways in which users might interact with the application. Not only does this leaves a number of negative test cases omitted from consideration. It can also induce problems, especially on major releases that contain new features and/or require user training.

As the development team matures, minimizing the effects of both confirmation and congruence bias means that testers must commit to more effort on differentiating and exploratory testing. This means that attempts should be made to break the application or system in such a way that anticipates how various types of users might break it. This begins with simple things such as entering letters into a numeric-only field but should go well beyond to behavior variation, permutations, and edge cases. Quality testing tools such as Functionize that significantly improve the ability of your team to plan and perform such tests.

Survivorship bias

Here’s a common misconception: If you want to achieve success, focus only on what is successful. The reality is that when failure is ignored, there may no longer be much of a distinction between success and failure.

Survivorship bias is a type of selection bias in which data set evaluation is distorted by focusing on the successful cases while ignoring the failures. It’s important to keep in mind that the failures may be ignored passively since the failures don’t persist or “survive” such that they could be measured. 

Testers are in possession of something that is silent. Or, rather, very quiet. But, it’s not advantageous that it remains quiet. The near-silence on the issue in question is likely the cause of some irrational practices that partially characterize poorly functioning test teams. We are referring to the silent catalogue of all the bugs that were discovered and kept from shipping with the release. These are all the potentially damaging defects that the team remedied before they could escape and do any harm. Following the release, no one gives them another thought. And why not? It’s as if they no longer exist, except perhaps as forgotten line items in a database on some dinosaur server. But there they remain, in the dark. Hundreds of them—voiceless yet indisputable evidence of disasters that never materialized, reputations that have been upheld, and money that was saved. 

Meanwhile, the software product is in the hands of the end users, exposing its inevitable and flaws, prompting sales teams and senior managers to vent, “What does QA do all day, anyway? How can they neglect quality like this? This kind of response is a classic example of survivorship bias. The presence of any errors induces them to undervalue the testing staff because they are oblivious to all the bugs that were successfully found and did not ship with the software. 

One way to mitigate survivorship bias is to convert all of the archived evidence into well-documented evidence and transparently publish a record of all bugs that are found in every release cycle.

Fundamental attribution error

This bias manifests itself when people tend to blame others for problems or issues—yet tend to be very accommodating or deflective when they are the source of similar problems. For example, if a developer inadvertently creates a defect, s/he is often seen as negligent by a particular tester. However, if that tester creates a defect in test code, that person might attempt to explain it away as the result of an upcoming due date, fatigue, or poor requirements definition. If an entire team culture exhibits this, then the atmosphere is likely to be contentious and divisive.

To mitigate the effects of this bias, a team must intentionally and steadily find ways to reduce the potential for more failures. For many teams, this would involve some level of test automation to ensure that the same problems don’t surface again. Testers and developers must also work collaboratively to take ownership of their work as well as their faults. This will help reduce the negativity and provide a path for quality improvement.

Negativity bias

A negativity bias is the tendency to give more attention or more consideration to what is conceptually negative, instead of maintaining balance with the positive experiences or information. Negativity bias is especially common in QA teams. Since testers primarily involved in find defects, a large aspect of their work can be perceived negatively. It’s all too easy for a tester to assume that a software feature is not shippable because of all the broken things that have been discovered.

This highlights the importance of the product manager responsibility for defining acceptance criteria together with the QA lead. There will be a healthy tension between the test lead that wants to maintain a zero-defect posture and the product manager who is likely to reject any acceptance criteria that requires an in-depth technical understanding. 

One of the exciting developments at Functionize is fully autonomous test modeling that is 100% based on live user analysis. By expanding testing coverage derived from live user interactions, our AI-driven platform is able to suggest workflows that may have been overlooked or assumed to be irrelevant, thus mitigating one source of algorithmic bias.

Recognition is the key

The authors of the book, “Lessons Learned In Software Testing, A Context-Driven Approach”, caution that “You can’t avoid these biases. They are, to a large extent, hard-wired into our brains. What you can do is manage the biases. For instance, just studying biases and practicing at becoming aware of them, you can become better equipped to compensate for them in your thinking. Diversity is also a protection against too much bias. If multiple testers brainstorm tests together, that can minimize the impact of any one tester’s bias.”

With respect to each type of bias, the most important effort is to recognize that they are likely to exist on your development team and among your stakeholders. It’s worth the effort to identify them, think about the context, and take steps to improve relationships or processes accordingly.