Why Automated Testing Enables a Project, But Doesn’t Always Get the Job Done: A Discussion With John Stevenson and Ray Grieselhuber – Part I

For more than 20 years, John Stevenson has been actively involved in software testing, primarily as a software tester. During that time, he has become an avid blogger (“The Expected Result was 42. Now What Was the Test?”), regular tweeter (@steveo1967) and author (“The Psychology of Software Testing”).

Passionate about software development, John has explored the theory and use behind automation and its role in testing. To gather more of his unique insight into automation, we chatted with John and Ray Grieselhuber of Functionize to discuss how it benefits testing and development, but doesn’t necessarily take the place of manual testing altogether.

Explain automation and how it doesn’t replace manual testing, but instead benefits it.

Ray:

In the companies that I’ve worked at, the biggest challenge we’ve had has always been everyone loves the idea of automated testing. Frequently, I run into trouble with actually making it happen because for the smaller companies I’ve worked with, typically, the view of the programmer is that there’s no guarantee that what they’re building at any given time is going to be the final product anyway. It’s almost like you’re writing prototype code as you run. So taking the time to write it and build the automation for the testing is something that just gets completely overlooked, unless you’re a really disciplined product team. That’s a challenge around automating testing.

The way I would define automated testing is basically as you are writing specifications for whatever piece of software you’re building the idea is that you are figuring out a way to code in the specification so that every time you add or make improvements to whatever software you’re building, every time you push new changes, it’s going to automatically ensure that that specification has been enforced and it can catch any bugs.

John:

That’s interesting because my mindset goals totally off a tangent on, especially if you’re trying to work from a specification. We’re now in a modern world where specifications can be very loosely coupled together. In the old days, good or bad, we had requirement documents.

If I take a step back on that and have an understanding of language, and it’s quite interesting how people do communicate and talk. If we take automate, it’s a word that’s taken from Greek, from automas, which means acting of itself. Without any interruption, I would guess, that’s my interpretation of it, with a human being, it acts on itself. That’s quite interesting.

When we’re talking about software automation, is it really acting on itself or of itself? At the current level of artificial intelligence within software at this time, I’m not sure we’re at a level where it could go beyond being told what to do, what it’s been coded to do. There is some aspects in big data analysis and at some level of AI, where it’s becoming what we would class as slightly more intelligent.

It then goes back into that debate, what do we mean by intelligence? This is where this whole battle line and divide has started to happen. No computer program up to now as far as I’m aware has passed the Turing test. No AI has passed the Turing test at all. That’s 60, 70 years now.

It’s quite interesting. I didn’t know this until a couple of years ago. The word computer is actually a term for a human being originally.

To quote Turing: “The human computer is supposed to be following fixed rules. It has no authority”, and this is the key between automated and testing thinking, he has no authority to deviate from them in any detail. Most automation systems we build, most automation we do, and this is why it won’t replace people and manual testing, is that it will not deviate from doing what it’s done before.

To go back a step, I like to use the definition that was done by James Bach and Michael Bolton— testing and checking. It’s been a big debate in the software industry, but I like the definition. Testing is what we do now. It’s what we class as the manual stuff. The checking is where a machine runs that behavior or that known, as you said earlier, the requirement part, the specification part.

When we have a test, you can class that as an instance of testing, and a machine could run a test and then it’s a check. It’s worth for people to go along and have a read of this article because it changes people’s minds. When you then start talking to others, and we all use the same common language, are we checking or are we testing? Are we writing stuff to do the checking?

Remember, you’re still going to need human beings to write that automation. This is the problem people find. Why don’t we just automate testing? Why don’t we automate coding?

Where should automated testing fit in your testing strategy?

John:

The biggest problem we face in the industry from a testing perspective, mainly from the testing perspective, is what is the value to the business of automating this? That’s the first question everybody should ask.

What I mean by value is is it going to change a lot? Is it going to be fragile? Is it going to be implemented and then thrown away? There’s the cost there. There’s the cost to maintain it. Once you have some sort of state, it lets you look at ways in which you can automate it that provides business value – that’s the number one priority.

I think a lot of that gets lost, and we say, we can automate this. We can automate that. We can automate everything along the line. I’ve probably stolen the words from somewhere, but I call it intelligent automation. Be intelligent in the automation you are doing. Don’t automate everything. Automate just enough to give you some level of confidence that your business value is protected. That’s it.

Ray:

You have to treat automation as an IT project, almost as a software development project if it’s doing right. You mentioned it will frequently get tacked on – not automation, just testing in general.

Basically, everyone has these ideas for a product, and we get excited about building new products. Testing by its very nature becomes an afterthought for so many people, especially people who don’t come from a testing or even an engineering background. They just want to get it out of the door as quickly as possible.

What happens, and it’s a good problem to have, is when the project is successful then you have people starting to use it. Then when things break, the inverse of that of course is if you don’t do it, you’re basically destroying the business value that you’ve created every single time something breaks. It becomes a branding issue. It becomes an issue of how people perceive your company and your product and the overall reliability of everything you do. It matters, and people realize that it matters big time far too late in many cases.

John:

Oh, yes. This is why I’ve now jumped on the bandwagon of Agile, but Agile is nothing new. Go back to NASA getting a man on the moon. I don’t know if you’ve hear anything about Gerald Weinberg. Highly recommend any of his stuff on “The Psychology of Computer Programming”, and the “The Secrets of Consulting”, any of his early stuff. They were doing iterative software engineering at NASA.

He’s what I would classify as the godfather of software testing. A lot of his stuff, some of it is nearly 40-years-old, is still current and valid. That to me is amazing. Hence, the name. You’ll probably see from the title of my book. I’m still writing in an Agile way. It’s published, but I publish a chapter at a time, which is “The Psychology of Software Testing”. It’s an homage to his book, “The Psychology of Computer Programming”.

Taking a step back to the fact that we’re now working in a “quicker iterations, get-to-market quick because we lose money, our competitors will get there first” mentality, it doesn’t mean you don’t need to test. You need to build quality from day one.

Testers should not wait until the end. This is the thing. We wait to automate. We wait to test until the end. Testing is day one, as soon as you start. First question testers should be asking, “Have we got a CI in place? No? Let’s build one together.”

Why does understanding the context of the testing you want to automate matter?

John:

Imagine you’re big data organization. You’re a tester in a team. You have to have a massive amount of data. Are you going to enter that manually? Are you going to use a tool to ingest the whole of that data quickly for you? I’ll use the tool because that’s the best use of time and the best use of automation to do that for me.

What I’m working on now, automatic deployment of components, it is a tool to help me and the teams I work with test quicker, to actually get in there, manually explore the system and look for the things that we don’t know about. Remember, automation can only deal with, and this is a line from Donald Rumsfeld, the known knowns. That’s all they can deal with.

Testing, on the other hand, highlights the unknown unknowns, and makes the known, knowns and known, unknowns. This is where people get confused. They get confused and think automation can replace all our testing. No. The machines are there to get rid of the boring rubbish, the stuff that we, in the old days, use to tick boxes.

Ray:

It occurred to me that every field, especially on the technology side, the temptation for probably both your average person in that career, and also for the business people on the outside looking in, is you push those people into a purely technical role and not really think about the big picture.

One of the things that I got immediately just from looking at the way you [John] structured your book, and true to the title of book, there’s a whole philosophy, a whole psychology, behind why these things matter. It’s almost taking a Renaissance man approach to these things where it’s far less about the technology and more about making the right decisions, understanding the strategy, and understanding the psychological reason we do the things we do.

Check back for part two of our chat with John and Ray as we continue to take a deep dive into automation. We’ll discuss the unique human behaviors manual testing offers, how testers roles are changing and where they see the future of automation heading.