ABCs of ROI and TCO for software teams

Decisions about what, when, and how to automate software impact a company’s bottom line, either adversely or positively. This primer for software testing and QA teams brings you up to speed on business concepts that affect your projects.

The decisions developers make about test automation and other aspects of software development can impact the company’s bottom line, one way or the other. You don’t need an MBA to use business concepts like total cost of ownership (TCO), return on investment (ROI), and opportunity cost. Yet understanding the terms can help you improve decision-making around whether, when, and how to deploy new tools.

By applying those business notions, you’ll land in increasingly abundant tech territory. “TCO, ROI, and opportunity costs play fundamental roles in software development, QA, and the continued product growth cycle as a whole,” contends Suren Sahaydachny, chief product officer at social media network startup Humanified. “I highly recommend properly conducting these diligences prior to the onset of the product or development processes, for smoother testing than you’d get by bolting on solutions on the back end of the process.” 

Plus, talking about business benefits in lingo that your managers use attracts corporate buy-in. Metrics like TCO and ROI help you make the case for any technology-focused project and getting executives on-board. 

With its rising adoption, test automation is a particularly ripe area for quantification in these common business terms. Custom software and web development company Dev.co, for instance, is using it for automated QA as well as regression testing. 

“Automated QA is definitely becoming the norm. It’s a lot easier for a bot to find issues and generally a lot more cost effective,” says Nate Nead, Dev.co’s CEO. “We typically use some hybrid of human and machine for QA testing. For all clients, we offer a quantification of both.” 

But to sell test automation to other departments, such as Finance, you need to show numbers. “Even though the whole idea of automation is popular among businesses right now, many are still skeptical about adopting it on a full scale,” says Eric Carrell of RapidAPI, an API marketplace for developers. “So we’re going to need supporting evidence to make the case for adopting technology at the cost of any forgone skilled personnel, among other factors,” Carrell adds. “Numbers and data that depict how favorable it is for a particular company to automate testing at its current levels helps in defending the move.”

Opportunity cost

“Opportunity cost is the cost of not performing Activity A because you are performing Activity B,” says Paul Grizzaffi, principal automation architect at custom developer Magenic and a frequent speaker at automation conferences.

Why do some companies measure opportunity costs? Because their resources are not infinite, explains testing specialist Catherine Powell. “There is never enough time, machines, or people to do every test we would like. Measuring opportunity cost helps us make the best choices given the limits that exist in the real world,” Powell says. “The opportunity cost of doing one test is the benefit we would have gotten from the other test we would have done.” 

In certain places, opportunity costs are measured in dollars as a matter of company policy, she writes  But clearly, given the choice, many software professionals would opt out of measuring opportunity cost in favor of ROI. 

“Our clients are always calculating an assumed ROI and analyzing the business viability of custom development, including QA. However, I think it's less customary to directly calculate opportunity cost,” notes Nead. 

Opportunity costs assume there’s an alternate, better opportunity for investment than the one you’re considering for ROI calculation, says Nead. “In the case of most of our projects, it's not the next best opportunity that is different, but determining which features will provide the most ROI-pop. It's not closing one opportunity for a better one. It’s determining which feature will be in version 2 versus version 3 of the product.”

TCO

Software TCO is aptly described as a combined estimate of a program’s direct (capital) and indirect (operational) costs. Factors involved in measuring TCO can include financial output, time, labor, and operational efficiency, for instance. However, benefits are left out of the picture; TCO only looks at the costs.

Indirect costs can vary tremendously by organization, industry, and specific scenario. But when it comes to test automation, one major aspect to consider is ongoing maintenance costs after implementation, Grizzaffi says. Costs in this category can include maintenance of tools, frameworks, and scripts along with updates to documentation, licenses, and tools’ support agreements. 

Organizations that start deploying test automation without in-house expertise may also encounter additional costs, such as hiring new staff, training existing staff, or paying consulting or outsourcing fees. 

Beyond that, test automation software must sometimes be customized to work with other development tools. Open source tools are available free of charge, but often lack traditional types of tech support.

ROI

Many contend that ROI offers a much more comprehensive approach to quantification than TCO, especially for front-loaded technologies such as test automation, because ROI weighs the costs versus the benefits.

An ROI calculation compares the cost of a project or investment with the projected value of its results to figure out whether the result is worth the cost. ROI is typically expressed as a percentage. An ROI of 30% means that a $100 investment will bring a $30 payback on the initial investment in one year. 

“Overall, automation is a bit cheaper, but the human element helps with qualitative issues and changes and change logs,” Nead explains. “The cost of implementing a true Agile test automation framework into software development teams is extremely front-loaded. However, for a team that consistently performs on software development tasks, we felt it absolutely necessary to invest early in these types of processes, as the ROI in the long run remained a no-brainer.” 

“I’d almost certainly use both ROI and TCO, but mainly ROI, in making a decision about automating an organization, as you can’t possibly go into a project like that without having an idea of costs and potential returns. I’d also use cost benefit analysis (CBA),” says Jack Zmudzinski, a senior associate at Future Processing. 

“While I do see TCO as the least important of the three metrics, I do use all of them in most of my decision-making processes,” adds Zmudzinski. “As TCO speaks only to costs, it should never be relied upon by itself for important business decisions.” 

To calculate CBA, you add up benefits and subtract costs. You can then use a CBA exercise to estimate and tally up the equivalent monetary benefits and costs, which suggests whether a project is financially worthwhile.

Current test automation cost benefits

Software and QA practitioners with test automation experience say they’ve already witnessed a range of measurable long-term benefits, including the following:

• Early bug catching. “When integrating test automation within the development, QA, and pre-launch processes, you’re going to catch more issues than would be found with manual testing alone,” Sahaydachny asserts.
• Lower spending on test development. “Automated tests can be reused multiple times or across multiple environments with few if any changes,” according to Andrei Mikhailau, director of software testing at global IT services company ScienceSoft.
• Faster results through parallel development. Points out Mikhailau: “Some tests can be run unattended, such as to verify nightly builds when testing goes on in parallel with development, or to test multiple devices or environments in parallel.”

Testers eye future benefits

Testing pros foresee the evolution of additional cost benefits for test automation in the future. “Test automation is being adopted as an industry standard process within modern development firms and agencies that are ahead of the curve,” according to Sahaydachny. “Test automation is often used in a supplementary context relative to manual testing, but as technology advances and ML/AI continue to develop, test automation will eventually become the sole reliance for testing and QA.”

“Ultimately, there will be technologies allowing truly fully functional automatic testing, for far more precise and comprehensive results than even the best manual testing today,” Sahaydachny predicts.

Need a primer on test automation? Here’s our primer to get you started.

‍

by Jacqueline Emigh

Jacqueline Emigh (pronounced “Amy”) is an award-winning journalist specializing in technologies used by enterprises, small businesses, and consumers. She has worked full time as an editor for TechTarget, BetaNews, and Ziff Davis. Her stories have also appeared in dozens of other major tech publications, including CIO, Linux Planet, and PC World. Jacqueline holds a B.S. degree in mass communications from Emerson College, with a journalism concentration. From 2017 to the present, she has served as a contributing editor to SD (Software Development) Times.

‍