Robot Framework: A Closer Look at Keyword-driven Testing Approach

A Brief History of Keyword-Driven Automation Testing

Keyword-driven automation testing paradoxically figured among the original solutions to the problem of scripted automation testing tools. Robot Framework led the field of open source packages. The solution of the time was to reduce scripting, not to eliminate it altogether. Perhaps surprisingly, these scripted testing tools required automation engineers long before the onslaught of Agile and Devops mandated engineers in QA.  Robot Framework presented a solution to the problem that early test scripts were either not reusable, or only reusable with significant complication because each new test case needed hard coding.

Robot Framework implements the method of “action words,” a set of keywords intended to make the arguments of certain test functions easier to invoke, and reduce the amount of coding required for new test cases.1 The keywords are moved out to a data file which is supposedly readable and more easily modified. Key functions are invoked by more user friendly language than standard programming languages. A function to click the login button after user inputs username and password is entered by a tester as, “Submit credentials.” And so here we see the predecessor of BDD scripting tools like Cucumber and its own scripting language Gherkin. 

A feature of heavily scripted testing tools is that user input to be tested in the SUT can be added to a list when the script used to run a test requires no change. This method of prescribing user input to modify the test scenario is also called data-driven testing to distinguish it from testing frameworks which require more hard coding of actual scripts. Ideally, the approach is easier to maintain and requires less technical staff and reduces the burden on engineering. But the problem today is engineers are still needed in QA. And web apps are getting more complicated and more difficult for mortals to manage. If apps are getting more complicated then it follows that the engineering skills required to test the apps will intensify. 

There is one extraordinary conundrum hovering around the rise of the myriad scripting tools intended to automate testing: if you can code Python then you can write your test scripts in Python; you don’t need to add another layer of tools. If you can’t program in Python then you probably can’t code Robot scripts either, because that requires Python, Robot’s Gherkin-like BDD syntax, and some knowledge of Selenese is implied although not absolutely required to follow what’s happening in the SeleniumLibrary (which must be co-installed with Robot). And Robot itself was written in Python! This appears on the surface to be a spiral staircase  of increasing development in QA. The conundrum is satisfied to a limited extent by Robot Framework’s report generation. Robot creates reports and logs in HTML form which can be easily reviewed in the browser, which is handy for testers already working on web app testing.

Robot’s Nuts and Bolts

Robot Framework’s intended objective is to automate acceptance testing, also called ATDD, or acceptance test-driven development. Tabular formatted data files contain the keywords and arguments for implementing test cases. The pivotal technology feature of Robot Framework is modularity. Libraries from many standard languages can be included and extended in this open source framework. SeleniumLibrary is actually required as a dependency of Robot Framework, and both require the Python interpreter.

Robot Framework, Python, and SeleniumLibrary plus its various dependencies can be installed with the pip package manager. Once installed, Robot Framework tests are data-driven, and use keywords which are specified in Test Templates and called by invoking their arguments in code. A theme inherent in Robot scripting, which is done in its own native IDE called RIDE, is the implementation of a workflow that is consistent with generic acceptance testing models. That is, Robot enables a repeatable pattern of testing which can be modified with new code changes. In practice, keyword testing divides the testing process into two phases: design and development plus test execution. Let’s look at a design and development example script provided by the architects. 

*** Test Cases ***               USER NAME PASSWORD
Invalid Username                 invalid ${VALID PASSWORD}
Invalid Password                 ${VALID USER} invalid
Invalid Username And Password    invalid whatever
Empty Username                   ${EMPTY} ${VALID PASSWORD}
Empty Password                   ${VALID USER} ${EMPTY}
Empty Username And Password      ${EMPTY} ${EMPTY}

*** Tester invoked keywords ***
Login With Invalid Credentials Should Fail
    [Arguments]     ${username}    ${password}
    Input Username     ${username}
    Input Password     ${password}
    Submit Credentials
    Login Should Have Failed

Login Should Have Failed
    Location Should Be     ${ERROR URL}
    Title Should Be     Error Page

Here we can see how keywords are used to invoke logic from coded functions to do things like actually clicking the submit button to sign in for testing a web app. There is a substantial community of users and many code samples and debugging tips are provided on repositories like Bitbucket and Github. A typical contributor article provides development tips on structuring tests, variable naming, and passing arguments, such as this: 

‍

Here we see an example script in which the dollar amount is passed as an argument to test a transaction. Some of the coding necessity is circumvented and replaced with more natural language components. But we also readily see how even this simplified Robot Framework scripting language quickly mounts up to a programming and development task, although this is the task it was ostensibly designed to reduce. And this brings us to a vista opposite the hyperbola, a less fortunate view of engineering tests which rival the development of the original software under test! Here is a simpler script which reveals an important problem with Robot Framework: browsers to be tested must be specified as in this code: 

robot --variable BROWSER:Chrome login_tests
robot --variable BROWSER:IE login_tests

‍

This looks like an old fashioned workaround, or hedging that must be done with old style test scripting frameworks. Although it may not yet be commonly known, this type of automation appears as a manual drudgery in the light of true automation testing, a drudgery that is no longer necessary.

Robot Versus True Intelligence

At the time Robot Framework’s inventor conceived the system as a Master’s thesis project in the 90s, truly automated testing powered by machine learning and artificial intelligence was a distant fantasy, not on the event horizon. The modularity of Robot Framework’s design enabled it to transcend platforms and become cross-technology compatible, and that was indeed innovation back then. That was cutting edge. But there is now a new caveat in the automation testing contract which is nothing less than a renaissance in the field: scripted automation testing is no longer viable. Scripted testing is deprecated, and machine intelligence based testing is now poised to replace all antecedents. Functionize leads this field with it's ML engine. It uses new methodologies in machine learning developed and extended uniquely by Functionize. One of the results are self-healing test cases.

Functionize’s testing platform not only removes scripting altogether and makes it possible to author tests very rapidly, it also contextualizes each step of the test creation so tests are no longer brittle. There is no need to write lines of code as above specifying various browsers. Functionize performs load testing simultaneously on hundreds of virtual machines and on all browsers. In Robot Framework, the browser that is used in the test case is specified by the ${BROWSER} variable which must be previously defined and scripted into the resource.robot file. Firefox browser is the default. Testing an app on other browsers must be coded. Functionize needs no such coding. 

Another irony in the lore of keyword-driven testing adherents is that test assertions can be abstracted in a way that makes them reusable. But in reality, they are only reusable when rescripted. Demos always begin with a login test and credentials, perhaps because this is a natural starting point. But it is likely the only assertion which needs little or no rescripting, and this is a suspicion lurking in the minds of managers looking for an escape from the burden of hiring engineers to write test scripts. Promotional language claims that keyword coded assertions are readable; but it remains contentious if only developers can read it. 

Functionize delivers the desired escape from an engineer dominated QA with a truly intelligent testing platform. Functionize is more like your testing assistant. Imagine a tireless and obedient assistant with a perfect memory for details, one that knows how to deliver software free of exceptions, an app in full compliance, an immaculate customer experience. 

References: 

https://blog.xebia.com/ 

https://bitbucket.org/robotframework/webdemo 

https://github.com/robotframework/HowToWriteGoodTestCases