The challenges of testing GUIs
As a matter of fact, you can deploy GUI testing. But it's not that easy, especially for mobile devices.
Sure, you can test GUIs. But it's not easy, especially for mobile devices.
Testing back-end code is relatively easy. You know what the software is supposed to do, so you write scripts to make sure that's what it actually does. Then you can run those tests as regression tests every time the code changes.
Testing graphical user interfaces (GUIs), however, is not so easy. There are far more moving parts, which inserts uncertainty, and that makes it harder to test the front end code reliably.
For example, you can record how a screen looks on a developer's computer, and later compare the bits on a test computer. But what happens if the test screen and the original screen have different dimensions, resolutions, text sizes, and color depths? The bitmaps never match, even when a human doing the testing manually would say that the test screen looks fine.
Beyond bitmap comparison
It used to be reasonable to compare bitmaps for GUI testing purposes, back in the days when there were only two possible video adapters for PCs and all applications ran full-screen. Today, trying to nail down baseline bitmaps for all the combinations is simply intractable. That’s due to the wide variety of display options, application windows that are sized at will, and most applications being displayed in a browser.
If we abstract the intention of GUIs, we can break the elements down into widgets, also called controls: menus, panels, text elements, input fields, check boxes, radio buttons, scroll bars, and so on. For QA purposes, we can specify which widgets should be in view for any given state of the program. Even that is complicated, as widgets also have a Z-order, meaning that one can be on top of another. Nevertheless, it's a more flexible approach than saving and comparing bitmaps.
Layout methods for GUIs may specify absolute locations and sizes for each element, or use a flow design methodology. It helps testers immensely to know which was used at design time. For absolute locations you can verify control coordinates and boundaries directly when you test; for flow designs you can access control IDs.
When you test GUIs with controls, testers have to ensure that for every possible user action (and there are sometimes thousands) every necessary widget appears. Confirm that each widget is correctly enabled or disabled for input given the application state; that the correct text appears and is not cut off; that disabled controls have gray text; that when one radio button is chosen the others are deselected; and so on. In an internationalized application, you need to repeat most of these tests for every supported language.
In addition to desktop applications, there are GUIs to test in desktop browsers, native mobile device apps, and mobile browsers. Unless you have tight control over your applications’ users, which is rare, you typically need software implementations for all the above environments, and you have to test them all as well.
That's especially onerous for mobile devices, as there are dozens of popular devices and thousands of total models of devices.
Testing tools
You aren’t left to your own devices… so to speak. While manual testing can be important – really, there’s a balance! – professional QA folks always look for software to take care of the tedious details. Quite a few testing tools aim to make this process easier, such as these two open-source options:
Selenium is a family of browser automation tools with an Apache 2.0 open source license. It underlies several open source and commercial browser-based testing tools. Selenium WebDriver can drive all major browsers and is recommended by the W3C. Selenium IDE is a Chrome and Firefox browser extension for record and playback tests in the browser. Selenium Grid takes WebDriver to another level by running tests on many machines at the same time.
Appium is an open source test automation framework for use with native, hybrid and mobile web apps. It drives iOS, Android, and Windows apps using the Selenium WebDriver protocol.
While there are dozens of commercial GUI testing tools, our sponsor, Functionize, combines natural language processing, deep-learning ML models, and other AI-based technologies to empower your team to build tests faster that don’t break and run at scale in the cloud. Unlike most other testing tools, Functionize doesn't require you to write scripts.
Crowdsourced testing
Another approach to GUI testing that's useful for testing on a variety of mobile devices is crowdsourced testing. About a dozen companies offer this service. The basic idea is that the companies offer a test platform and a pool of testers with diverse devices, languages, and locations; you provide the application and scenarios, and you generally pay per bug found.
Crowdsourced testing and internal testing are complementary, and together give you better coverage of your code. Crowdsourced testing is best for late-stage uses such as beta and compatibility testing, while internal testing, with and without tools, is useful throughout the application lifecycle.
Overall, yes, it is possible to test GUIs. You should combine all of the techniques we discussed: suites of back-end regression tests, manual GUI testing, automated GUI testing with tools, and crowdsourced testing.
Automated testing helps you deliver reliable software on time and on budget. But test scripting has its limitations – and naturally, at Functionize we feel that we’ve fixed those problems. Let us tell you a little more about how to improve code coverage.
by Martin Heller
Martin Heller is a freelance writer. Formerly a web and Windows programming consultant, he developed databases, software, and websites from 1986 to 2010. More recently, he has served as VP of technology and education at Alpha Software and chairman and CEO at Tubifi.
