5 Roadblocks to DevOps Success

DevOps success is about delivering reliable, high-quality services to your customers. Central to this is ensuring that new features can be released without breaking anything. However, the current approach to testing creates roadblocks to DevOps success.

DevOps success is typically measured by things like velocity (deployment time for new features), quality (how stable are the new features and do they cause downtime) and reactivity (how quickly can the team recover from an incident). In the perfect world, your DevOps team delivers changes seamlessly as soon as they are developed. Customers see no disruption to their service during this process. Any incidents are handled seamlessly, with no customer downtime. But this ideal world is hard to achieve. There are several major roadblocks to DevOps success. Here, we look at 5 of these and show how an autonomous intelligent test agent can solve them.

How to achieve DevOps success?

To achieve DevOps success you need 3 things. Firstly, you must have good test automation that can regression test your entire system quickly and efficiently. Secondly, you need to be able to integrate your testing with your deployment system. In other words, testing must become an integral part of your CI/CD pipeline. Thirdly, you need to be able to assess the impact of new features on your existing infrastructure. This is usually achieved by employing techniques such as canary testing, feature flags, and instrumentation. The following diagram shows how the process should go.

Roadblocks to DevOps Success

However, there is a problem with all this. Standard test automation simply isn’t good enough to achieve this. In other words, it is a roadblock to DevOps success. So, let’s look at the reasons why.

1. Test creation

DevOps success requires your entire regression test suite to be automated. In other words, you need 100% automated test coverage. Unfortunately, using Selenium and related scripting tools, this is nearly impossible. There are several reasons for this. Test scripts take ages to write. Each script has to then be optimized and debugged to run cross-browser. And the real killer – dynamic content and 3rd party plugins are almost impossible to test properly.

Each test script is a mini software development project. A simple script may only run to a few hundred lines of code, but this has to be created, tested and debugged by hand. A script is simply a list of actions that replicate a user interacting with your UI. For instance, selecting an item, adding it to the cart, and then going to checkout. The script knows which element to choose based on selectors. The reason it takes so long is the test engineer needs to find the best selector for each element that is involved in the test. They then need to add checks on each screen to ensure things loaded as expected. Realistically, a good test engineer will take most of a day to create a simple test.

2. Handling modern UIs

Having created the test, you then have to optimize it to work cross-browser and cross-platform. The problem is, modern websites are rendered dynamically using libraries like React, Angular or Parsley. These can lead to all the element selectors changing between browsers. The result is, the test engineer has to manually adapt the test for each browser and platform. This adds additional hours of work.

Modern UIs are generally dynamic and responsive. They also often incorporate 3rd party plugins. Dynamic content can cause problems with selectors since many of them will change dynamically. Responsive sites mean that elements may load in a different order, which can result in the wrong element being selected. But the real killer is embedded content. Because this exists in a nested DOM (domain object model), elements within it can’t be accessed directly by Selenium. The only way to access them is to construct complex XPath queries, which takes real skill and eats time. 

The upshot is, creating just 100 simple tests is 3 months' work. But since most regression suites consist of thousands of tests, few teams even achieve 50% test coverage. In other words, half of your tests still have to be done painstakingly by hand. Not ideal if you want to automate your release pipeline for DevOps success!

3. Test maintenance

One of the biggest nightmares for test engineers is test maintenance. Put simply, every time your UI changes, many of the selectors will change. As a result, most of your tests will break and appear to fail. These tests then have to be manually debugged by your test engineers. Anecdotally, this process often consumes half their time.

Let’s look at a simple example of why this happens. Imagine that you want to restyle your site. You now add a drop shadow to your buttons and add some classes to your CSS. A simple change, you might think. But the problem is, this change can cause CSS selectors to change. If your scripts are using those, then they will break. Or another example. What if you now add a new menu entry in your menu bar at the top of the page. Now, your scripts are probably going to click on the wrong entry, again causing real problems. 

Clearly, this has a massive impact on DevOps Success. Even if you have automated all your tests, every change triggers the need for massive amounts of test maintenance. So, you have immediately lost the benefit of the automation. Where releases are done at long intervals, this is less of a problem, since your test team can work on the maintenance during the release cycle. But with continuous releases, it becomes a major roadblock.

4. Test analysis

Automated tests are great. You just get told Pass or Fail and can be sure everything is fine, right? Wrong! The problem with automated tests is they can only test elements that have been explicitly listed in the test script. Unfortunately, time and efficiency mean there is no way to test each and every element on a page. Just look at even a simple Google Search page. As you can see, there are dozens of elements, many of them concealed in dropdowns or menu grids. A really thorough test would check that every one of those items is there. But to do so would require hundreds of additional lines of code. So typically, your test script just verifies a few items on the page, and manual testers do a check to verify everything else.

Clearly, this makes automated release harder, unless you are prepared to risk some elements being absent in your UI. That, in turn, means you are going to put your DevOps success at risk.

5. Integrations

A key requirement for DevOps success is being able to integrate all the tools into a single release pipeline. This allows releases to be done smoothly and automatically. More important still, it makes it easier to roll back any changes that cause a problem. Unfortunately, Selenium by itself isn’t that easy to integrate. Typically, the tests are run on stand-alone infrastructure or on a 3rd Party cloud. This means your DevOps team is going to spend their time writing custom scripts to try and trigger tests automatically. Some 3rd party tools, such as Sauce Labs, do offer integrations with more popular CI tools (e.g. Jenkins). But even then you will need to install additional plugins and perform a long-winded setup. All of which eats into your DevOps team’s time.

How an intelligent test agent helps DevOps success

By now, you might be feeling a little disheartened. It may seem that DevOps success is forever outside your grasp unless you have unlimited resources to throw at the problem. But despair not. There is a way to ensure DevOps success and to maximize your test automation. The answer is to turn to Functionize’s Intelligent Test Agent (ITA). This ITA uses a whole range of artificial intelligence (AI) approaches in order to solve the problems of test creation, analysis and maintenance.

Natural language test creation

Functionize’s Natural Language Processing takes structured test plans and unstructured user stories and converts them into functional tests. It does this by using the test plans to build a model of how your AI works. These tests will automatically run on any browser or platform. And they are also able to cope with embedded and dynamic content. This means achieving 100% test coverage is as simple as writing the required test plans.

Visual test analysis

Every time you run a Functionize test, the system records screenshots before, during and after every step. These are analyzed by an intelligent computer vision system to identify things that have changed that shouldn’t have. The system is clever enough to ignore dynamic content (e.g. the date, newsfeeds, etc.). But if it spots a change, this is flagged to the user and highlighted on the image. This means our tests cover the entire UI, not just the items you specify in your test plan.

Self-healing tests

The Functionize system is constantly refining its model of your UI. It uses a combination of machine learning, computer vision, and natural language processing to allow it to learn how your UI works. This allows it to cope with changes to the UI such as elements moving on the page, buttons being renamed, and style changes. Some changes are more subtle and cause errors many steps later. Root cause analysis allows it to track back to identify the most likely problem, find and test a solution, and then present this to you for approval.

Integrations and the Functionize Test Cloud

Functionize tests all run on the Functionize Test Cloud. This offers you dynamically scalable capacity, the ability to test mobile as well as desktop, and stores the results of all past test runs. The system offers simple integrations with all CI/CD tools, making it easy to add it to your release pipeline. The upshot is, you will be able to conduct your full regression test suite in just a couple of hours. An essential aid to DevOps success.

Conclusions

There are many things standing in the way of DevOps success. But if you start to use Functionize, you remove a large number of the roadblocks. Now, you will be able to release with greater velocity, better quality, and with less risk of impacting existing customers.