User Acceptance Testing

In the history of software, there have been many classic examples of well-executed code that is so poorly designed for users that it ended up alienating the user base or became a complete failure. Some of you may remember “Clippy”, Microsoft’s infamously bad Office Assistant who was almost universally hated by users. This was an example where a company thought they knew what their users really wanted but failed to test it properly. And how about Apple Maps, proving that even the kings of User Experience can get it wrong. Launched in 2012, it featured missing towns, wrong locations and poor search results.

These both represent failures of one of the most important aspects of Software Testing, namely User Acceptance Testing (UAT or UAT testing). This is the final stage of the software testing process, coming after you have completed your unit tests, exploratory tests, smoke tests and regression tests. Done properly, UAT will ensure you deliver the software your customer wants and will ensure you can pay the bills. Done badly (or not at all) and you risk disappointing your customers or worse. 

What is UAT 

User Acceptance Testing is the process of verifying that your software achieves the task it was designed for, in an easy-to-use fashion. Or as Wikipedia puts it: 

User acceptance testing (UAT) consists of a process of verifying that a solution works for the user. It is not system testing (ensuring software does not crash and meets documented requirements), but rather ensures that the solution will work for the user (i.e., tests that the user accepts the solution); software vendors often refer to this as "Beta testing". 

While the above definition says that UAT is synonymous with Beta testing, in my view there is actually a subtle difference. For one thing, in beta testing, you are also aiming to test things like the scalability and stability of your backend under real-life load. For another, all too often, companies undertaking beta testing fail to properly solicit feedback from users, preferring instead to rely on passive techniques such as analytics. 

When to do UAT? 

Before you are ready to start UAT you need to make sure your software has passed all the stages of systems testing. Essentially, your QA team needs to assure you that your code is bug-free (or certainly free from any bugs that affect the functionality). UAT is definitely not designed to identify technical bugs and if these emerge during the UAT phase, you should look to improve your testing process! 

UAT also needs you to work collaboratively with the users. Where there is a specific client, you will need to schedule the testing with them. Where there are multiple users you will probably need to plan and launch a beta version of your software with a group of active test users. This will need careful planning and management, along with buy-in from people like the Product Manager, Head of Sales, etc. 

Why do UAT? 

UAT has two principle aims. The first is ensuring your software meets (or exceeds) the user’s expectations of what the software should achieve. The second aim is to ensure the software is actually usable and useful. This means that it avoids classic pitfalls such as incomprehensible error messages, poor help text or key functions that are buried at the bottom of a hierarchy of menus (think Windows 8 shutdown). 

Your choice of software development model can have a big impact on how easy the UAT process is. In the classic waterfall model, the product is fully defined before any development begins. As long as that process was done effectively this should make UAT reasonably straightforward. However, all too often, it’s only once the software is delivered that the customer starts to understand what they really wanted. This was one of the big motivations behind the Agile Development Manifesto. 

Acceptance Criteria can help here. Acceptance Criteria are a part of User Stories, but also exist in other forms of software specification. A well written set of User Stories will give a clear list of what the software should achieve, along with a list of criteria that need to be passed in order for the software to be accepted by the client (the so-called Acceptance Criteria). But whichever model of software development you are following, you need to have a clear definition of what the software was trying to achieve, a way of assessing when the software is complete and a good understanding of the stakeholders. Which brings us on to… 

Who are the “users”? 

For many software development houses and web development companies, the user is the client who is paying for the project. In these cases, it is essential that the user is completely involved in the process of testing. They may have taken part in regular demos and meetings during the actual development (especially if this was an agile project). But when you reach the UAT stage you will need to be more formal, as we will see in the next section. 

But what if you are a business that sells direct to consumers, such as an app developer or a games studio, or even a vehicle manufacturer? In these cases, the user and other stakeholders are probably going to be represented during testing by the Product Manager, possibly supported by the Head of Sales. Real users will be involved, but their involvement may be through downloading a beta version of the software or may consist of taking part in coordinated user panels with the Product team. In both cases, the views of the users are central to the process of UAT.  

How to do UAT? 

The process for UAT is very similar to other forms of testing. What differs is that the criteria being tested against may not be purely technical. For instance, usability testing will involve a large amount of subjective assessment and may require things such as A/B testing to choose between alternative approaches. 

As always with testing, the process can be broken down into 3 phases. Plan, execute and review. During the Planning phase you need to capture all the requirements you are testing against. You also need to identify your users and either plan a beta launch, or a series of tests with the end user. This is also the stage where you design your test cases. These may be as simple as a list of tasks to get the user to complete, or you may be comparing two versions of a UI feature to determine which is easiest to use. Most UAT tests will follow the black box approach. What matters isn’t how the software works, just that it delivers the expected results for the user.  

During the Execution phase you need to ensure you carry out all your tests and solicit useful feedback from the users. This may be in the form of questionnaires or f2f meetings, or it may be through instrumenting your software and analyzing the results. Any bugs or UI improvements identified need to be fixed and the tests re-run. During the Review stage you need to verify with the users that the system is fit for purpose and ready for production/release. 

The difference between Beta Testing and UAT 

As mentioned above, while UAT is often synonymous with beta testing, there are some subtle differences of emphasis. During beta testing you should be concentrating on three things:

1. Usability. This is a key aspect for any software that will have real users. This is a major aspect of UAT and, as mentioned above, can be one of the hardest ones to test due to its subjective nature.
2. Robustness. Software is complex, and few pieces of software are completely robust in the face of real users. Users have an amazing ability to find new ways to break things. Often this is tied up with usability – it’s not that the user is trying to break things, just that they aren’t following the assumptions that the developers made.
3. Scalability. Up to now, most of your testing will have been done in dev and staging environments. Even if your QA team has done extensive load testing, the chances are this has been done in a very mechanical fashion. You may be confident that your database can handle thousands of concurrent requests, but what happens when these requests are coming in from real users?

Conclusions

As we have seen, UAT is arguably the most critical stage of testing, yet often it is overlooked. Integrating UAT into your testing methodology is straightforward but needs some effort. However, that effort will bring with it three key benefits. Firstly, you will end up with happy clients and users. Secondly, if done properly, UAT can help you design your software more efficiently by making you focus on what the user really needs. Thirdly, UAT will reduce the time and effort needed to firefight unexpected issues further down the line. All in all, if you're not doing UAT already, you should be!