Why complete Automated Coverage is actually a bad idea

Wouldn’t it be nice to have 100% automated coverage of a test system? Imagine it. Not having to do any manual tests and every time something is updated, get a full report of the whole system. Everything would be perfect, no bugs would be missed. Right?

Sadly, no. This is impossible, that’s a complete utopia. There’s no way to account for each and every single one of the actions that a human can do in our apps. Why? Because human beings are unpredictable.

This is even more noticeable when we’re testing the UI with automated frameworks such as Selenium, Appium, and other front-end solutions for automated testing.

Avoid coverage for the sake of coverage

Even if the intention of trying to automate everything is noble and positive, sometimes that can lead us to write impossibly complex code that will be hard to maintain, fragile, and not cost-effective. To test unpredictable scenarios, we should use unpredictable methods (humans).

Where does the automation come in handy? With things like Regression Tests, Smoke Tests, and Performance Tests. The type of testing that always follows the same path and tends to be repetitive. Having that QA effort automated allows the human testers to focus on the fun stuff, which also keeps their concentration up and makes them more productive.

A good balance between Manual and Automated Tests is key in every QA Plan

How many manual tests vs. automated tests you have are up to you. That depends a lot on the type of system that’s being tested. There are systems that allow for bigger automated coverage than others. Sometimes if something seems too hard to automate, it’s because it shouldn’t be automated at all.

When our goal is to have 100% coverage of automated tests in our system, we’re very likely to run into these problems:

• Code that’s hard to maintain
• A Solution that seems to “have no end”
• Limitations of what our tools can do
• Slow progress (if any)

So, does that mean that we should focus only on covering a small portion of our system? Absolutely no. Not focusing on 100% coverage doesn’t mean that we should just cover the basics. What it means is that we should focus on adding tests that add value to our solution.

How can we decide what to automate, then?

There’s no golden rule for this, most of it falls into common sense. But usually, the best candidates for automation are:

• Smoke Tests: If your smoke tests are automated you will always have a quick way of knowing if your app is stable enough to proceed with the rest of the tests.
• Regression tests: A Regression Test Suite can get huge (and boring) really quickly. Automating this part of your tests will allow you to focus your QA Resources on testing new features with more concentration and less fatigue.

What not to expect from automated tests?

We should never expect to test new features with our automation. The first time a new feature is made is preferable to have actual eyes looking at it and testing it from as many sides as possible. Automated Tests are good for validating if something is still working as expected despite the changes made to the app. Not the other way around.

It’s a lot easier and more productive to write an automated test for a system that’s stable at the moment. A new feature cannot guarantee this. In an ideal world, a new feature is tested and documented by a human, and then, when that feature is tested as part of the regression tests it is automated based on the findings of the manual tests.

 

In conclusion, keep it simple, keep it tidy, and keep it meaningful.

This article was written by Mike Arias
Senior Software Testing Engineer of TechAID.
Twitter: @theqaboy
Blog: qaboy.com/