Should you build or buy a test automation framework?

In this article, we survey many of these factors and guide you in the approach toward making a build-versus-buy decision regarding a Test Automation Framework.

It’s a common question facing teams that are considering how they can make real progress with software testing automation. As you might guess, the answer depends on the specific testing needs of a team, the type of product that is under scrutiny, and a number of other factors. When software folks mention Appium or WebDriver or Appium, they are talking about software testing tools. These tools employ libraries and applications for creating, running, and storing tests. The framework is one level up from this. A framework can manage and execute many tests, optionally on a schedule, and generate meaningful reports of all of the results. Many frameworks track the history for all test results and provide the ability to view trends or statistics.

Where to begin with frameworks

WebDriver and Appium are software testing tools—applications and libraries for creating, executing, and storing tests. Some teams build elaborate test automation frameworks with these tools.

Some frameworks produce output that is consumable by other software, such as a continuous integration (CI) platform. Other frameworks provide dashboards so that humans can view progress and manage testing assets. Some integrate multiple types of testing approaches. Some of these are manual and many that are automatic. Manually running and interpreting test results functions can help your team get going quickly, but any serious test tool integration effort that extends beyond a few months is likely to benefit from the implementation of a test automation framework.

Should your team purchase an automated testing framework, or build one? 

The answer to that question must address cost, customization, and control. These are the three most important criteria for most teams to consider when working toward a decision. Before looking at each of those criteria, let’s start off down the right path.

Considering the cost of “free”

Because it’s open-source and the license cost is zero, Selenium is perceived to be initially quite attractive in comparison to the monetary costs of commercial test automation solutions.

But there are many other considerations beyond how much money will be spent on a subscription. The intangible, unforeseen costs of implementing an automation solution can eventually become much more burdensome.

Other very real costs include the following:

• While open-source tools and libraries provide a basic foundation, it remains for your team to do the hard work of writing code for test creation, execution, handling errors, test case management. You’ll also need to configure reports.
• In addition, your team will need to build out all of the test logic that it wants to automate.
• Many teams find it necessary to test integrations with external systems and back-end systems (through one or more APIs). These teams will need to add support for all of that testing into the new framework.
• For those teams that are considering a continuous testing solution, there will be additional effort to integrate with DevOps or CI/CD tools.

As you can see, all of this effort actually becomes an additional development effort. It will be necessary to create another team and assign another lead to manage this team. Because it’s Selenium-based, you’ve got to find skilled developers that are capable of building and maintaining the new testing solution. Some project management oversight is another cost that goes into the calculation.

The other costs of freemium

Don’t forget the costs of constructing and managing the infrastructure on which you run all of the tests that the team will build.

Upfront, it’s important to face the reality that these costs will increase when it becomes necessary to scale the infrastructure. The need to scale arises naturally with increasing test volumes, test performance, and parallelization.

Likely, you’ll want to run tests across multiple browsers and devices. So, be sure to account for those additional costs. Costs will also increase as you build out that part of the infrastructure. One option is to subscribe to Selenium-on-the-cloud service providers such as BrowserStack or Sauce Labs. 

Clearly, there are a number of initial and ongoing costs that come with building a Selenium-based test automation solution. In addition, keep in mind that there is a fairly steep opportunity cost to all of this. Since it will be necessary to employ development resources to build the test automation framework, those costly resources won’t be working on product development. Take all of these costs fully into consideration. Then you’ll have an accurate measure for the total cost of ownership for building a custom test automation solution.

Finding good value in commercial test automation solutions

The other side of the coin is the total cost of procuring and deploying commercial automation software that have license or subscription costs.

A number of commercial test automation frameworks provide an array of features that make it much easier to automate testing tasks. With the best commercial offerings, it isn’t necessary to employ developer-level skill sets to create and manage automated tests. Most of the automation can be done by anyone on the team many. Not only can this lower costs, but it serves to refocus developers on building product features—instead of tinkering with test automation code.

Most commercial automation solutions require that building and managing your own test infrastructure to run the solution. This adds to the total cost of using these solutions. Others, such as Functionize, provide a cloud computing solution that significantly reduces all infrastructure costs. Compare this with all the costs and tedium of home-grown automation. 

It's possible to have the peace-of-mind that comes with a limited, predictable cost structure for test automation.

What’s the real cost?

Commercial subscriptions and licenses typically include upgrade costs. This eliminates the cost of maintaining the framework itself.

If it becomes necessary to scale the test automation framework, then these costs will go up slightly. The good news is that most test automation vendors offer volume discounts.

A good understanding of total ownership cost is important before you make a choice of a test automation solution. Embarking on the effort to build a custom automation solution may be sensible if you expect to maintain only a small, rarely-changing test suite. However, expanding requirements., testing needs, and complexity may bring your team to the point of reconsidering a commercial solution.

The ability to customize

As you consider which direction to go, you’ll also want to consider the extent to which you need customization. This is a common concern with organizations that maintain special application features, integrations, or workflows.

Most commercial test automation software tools provide some type of support for customization. Since this customization typically requires a higher skill set, remember to factor this cost into your calculations. The upgrade path for commercial software is another important factor to consider if you need customizations. Be sure that any customizations that your team implements uses a stable, well-supported API that won’t interfere with upgrades.

More control with custom-built automation

Let’s give credit where it’s due. With a custom-built Selenium framework, the testing team has complete control over the features in the framework.

Of course, it’s necessary to spend the effort and money to build such features.  Conversely, it’s important to realize that the only features in this custom framework are the ones that your team builds.

With a commercial automation solution, there’s typically a heavy dependence on the tool vendor to add and maintain additional features. While most vendors don’t charge extra for such updates, there is little or no control over the vendor release schedule. Then again, there is a great benefit that comes from the collective feedback and feature requests from the customer base. Your team can gain from the experience and innovation that arises from that large group of your industry peers.

Making the choice to buy

Commercial tools have at least two major advantages.

Firstly, getting to some significant positive benefits happens much more quickly in comparison with a comparable home-grown approach. This happens even faster if the app your testing runs in a browser. For browser apps, setup usually entails a one-time configuration and perhaps a few adjustments.

A second major advantage with a commercial solution is the predictability and reliability, together with vendor support. Typically, it’s much easier to create, maintain, and update automated tests. Another advantage of commercial tools is the ability to generate test reports at various levels of detail. Many vendors provide pre-configured reports that provide charts, graphs, and historical data. 

These capabilities are quite uncommon or non-existent in open-source frameworks. The effort to build equivalent functionality would require excessive effort work to build. The trade-off is that commercial tools typically provide limited customization.

Building or buying a test automation framework

There can be no doubt that the future of automated testing is moving to the cloud, and Functionize is leading this movement. Not only will your company enjoy significant test infrastructure cost savings, but your team can leverage a robust test automation environment.

Our AI-based test automation framework can help you make better use of your testing resources. Many teams using the Functionize solution today have gone well beyond their previous home-grown stage. Get in touch with us today. We’ll help you understand how you can slash your test costs and dramatically increase your testing effectiveness.