End-to-End Testing Tools

Software development in the year 2018 is mostly repetitive tasks, and the bulk of end-to-end testing tasks is now scripted. Combining myriad consoles and scripting languages to achieve CI/T/CD was until recently state-of-the-art automation testing. Ultimately, this craving to automate everything motivates the creation of end-to-end testing tools. In reality, there is no single tool which satisfies the craving. But why automate a task when it really needs intelligence? If a redundant process can be scripted then we can use machine learning code to truly automate it. To reach this peak, we must first understand the reigning development regime and the tools we currently grapple with. Before we can make an intelligent change we must map this imaginary realm in which developers script the operation of myriad other scripts.

Automation testing of today is largely a developer’s task of writing programs to test other programs. This endeavor spawned hundreds of tools like Chai, Mocha, and Protractor. Today, an industry begging for a singular end-to-end solution to the piecemeal jumble of testing tools latches onto an amalgam like Cypress. But these supposed end-to-end testing tools really just mash all the Cucumbers and Gherkins into a distribution. They’re still the same tools and you still have to learn three scripting languages. It is not a homogenous tool, and it’s definitely not the solution. Functionize illustrates that intelligent software can completely replace both the redundant process and especially the scripting of test cases. In order to fully understand how, let’s explore today’s model of end-to-end testing by looking at the most popular tools and methods. First, here are the quintessential concepts. 

When a developer adds or updates a module the new file must be copied to a server accessible to end users. This was once as simple as using an FTP client to send an HTML file to a server. The page went live instantly. But now we live in the Amazon era, wherein millions of dollars are packed into a single button click, careers are built and demolished by mouse hover events, and this previously simple task is now a “pipeline” of associated tasks in which many agents intervene to ensure everything functions precisely as intended. Enterprises are now burdened to staff QA with engineers. One recurring task is to assign the correct permissions to the new file and to make it executable. As we dissect this pipeline, which is now described variously as continuous integration, continuous testing, and continuous deployment, we see layer upon layer of increasing complexity. A hodgepodge of goofy names like Jenkins and Groovy are no less mystifying to spectators than the engineering practices they implement. 

Jenkins scripts contain embedded Groovy code to automate the continuous integration of new modules. Bash scripts attached to code releases and called “hooks” in the Git repository lingo set permissions for the new file and do other file management operations. When it comes to testing the new module, the setup may be scripted with Cucumber, which is described as a Behavior Driven Development cycle (usually BDD). Cucumber’s own scripting language is called Gherkin, and is supposedly comprehensible to “business facing” team members. Decipherable may be a more appropriate word, but the prospect is spurious. Most of the testing frameworks include event recorders which create scripts as testers enter assertions. Coded UI is one such framework. Its Test Builder writes scripts compatible with .Net framework. Testers can then use the Visual Studio Enterprise edition to modify test cases. And the list of buzzwords continues throughout continuous deployment. We will have to delve into Docker and even get into Git if we are going to sort out this devolution before the road bends toward the abyss. 

The Rise of Repos 

All tools ultimately begin and end with the extraordinary rise of the versioning repository which has become standard fare in software integration, testing, and delivery. GitHub enables the Agile team to collaborate in new versions of code, perform rollbacks, and track changes to their apps. The core of this functionality is a set of event-driven actions attached to each commit. Events are scripted to react and automate every code commit. These events are called “hooks” in the Git realm. On each commit, Git checks the hooks directory to find and execute any attached scripts. Scripts run before, during, and after to correctly establish file permissions and deploy code, for example. This is a brilliant method of standardizing the deployment to ensure consistent compliance upon every event. Many end-to-end testing tools build on this repository strategy for sharing code and supporting documentation and data. 

All Your Cukes In One Basket? 

End-to-end testing tools have the purpose of testing the user experience and accuracy of an application, especially focusing on UI elements. During e2e testing data integrity must be verified, components and dependencies confirmed, and all discovered issues reported along the way. Although testing is straightforward in concept, tools which purportedly automate the process deliver limited success at a high overhead cost. The main source of this overhead is that fact that each testing tool is by nature limited to testing one target technology, but there are many technologies to test. The discussion begins with frameworks. 

Jasmine and Mocha are two competing popular JavaScript frameworks and have a curiously asymmetric set of strengths and weaknesses which tends to suggest, “we need to use both of these but at different times.” Jasmine has an assertion library, but Mocha does not and uses Chai’s assertion library. On the other hand, Mocha has a command line interface for prototyping script, but Jasmine does not have this nifty feature. Mocha is a Node.js based JavaScript framework which creates test coverage reports. Mocha and Jasmine both support asynchronous test case builds in a variety of browsers. In this category we also find tools of varying facility and popularity such as Karma and QUnit. When the challenge of choosing a JavaScript test framework is over, it’s time to figure out which assertion library is right for your test cases and app. 

Chai and Expect are competing assertion libraries to include in your JavaScript testing framework. These libraries support Behavior Driven Development (BDD) cycles and contain vast functionality for issues in UIs. Chai includes an assertion API which supports several styles of assertion including Should, Expect, and Difference. Chai is also extensible by way of a vast number of available plugins. A very basic Chai assertion is scripted like this: 

var john = new Assertion('John Brown');
john.assert(john._obj === 'John Brown', "expected #{this} to be 'John Brown'", "expected #{this} to not be 'John Brown'");  

Which leads smart customers to this new assertion: 

i.should.be.a('Functionize user!'); 

‍

Bare Metal Versus Nonmetal 

We are now in the middle of the automation testing abyss, and we are at the point which most developers face sooner or later, which is the point of realization that effectively all JavaScript based testing frameworks require Selenium. Here we have yet another Apache success story. The playback functionality of Selenium enables testers to record, rewind, and replay test assertions. The IDE also supports coding a wide variety of languages like Python and Groovy. There is also WebDriver, which is Selenium’s browser interface. Selenium Grid also enables running test cases on remote servers through virtual machines. Now another membrane to permeate is the open source versus paid testing framework choice. MS Coded UI falls toward the expensive end of the paid testing software spectrum, and there is nearly no community support online. Like with most MS products people who use this one amount to a captive audience, resulting from upper echelon enterprise decisions. 

Wrapping it all up: 

The next step in our end-to-end testing tool journey is to choose a wrapper for Selenium. What does this mean? Generally it means that we need another tool, like protractor or nightwatch Protractor is indeed a wrapper, because it actually contains Selenium Webdriver! And Protractor is also billed as an end-to-end test framework especially for AngularJS applications. Finally, we will need some more libraries like Sinon and TestDouble, because Mocha really does not include these doubles. And that brings us to the exciting conclusion of 101 testing tools (that is the number of tools, not the MOOC title). But there is still one monster tool which we must not overlook. 

And now for the denouement, now for the clencher, there is one tool which boldly promises to be truly end-to-end, and not just front-to-middle or off-to-one-side. Cypress claims to do all of the above and without Selenium. It supports mocking and stubbing to boot. But there is one curious reality: Cypress is actually a collection of open source tools, one from each category described above, mashed together for your convenience into a huge distributable: like Anaconda to Python, but perhaps on a smaller scale - and packaging relabeled freeware products together is all too common today. It begins to feel like Cypress’ whole existence is a Selenium weak point orbital diagram. Their copy reads, “Cypress tests are only written in JavaScript,” as if a JavaScript coder can’t read “Selenese.” That is ad copy intended to persuade. 

QA Developers? 

Instead of moving toward intelligence in automation, testing now requires an overhead of scripting by QA Engineers with advanced degrees. Look at Jenkins, for example, a common skill requirement of Fortune 500 QA engineering team candidates. Jenkins is an automation server, which runs on Apache Tomcat servlets, and which enables developers to script a lot of continuous integration. This current trend toward engineering level testers is not viable because customer experience is ultimately defined by customers who cannot be likewise required to have engineering skills to operate the application under test. Now we are in position to bring intelligence to replace scripting. Functionize brings a unique intelligence to replace the scripting that once encumbered engineering talent, allowing them to optimize their focus on product development.