Why complete Automated Coverage is actually a bad idea
Wouldn’t it be nice to have 100% automated coverage of a test system? Imagine it. Not having to do any manual tests and every time something is updated, get a full report of the whole system. Everything would be perfect, no bugs would be missed. Right?
Sadly, no. This is impossible, that’s a complete utopia. There’s no way to account for each and every single one of the actions that a human can do in our apps. Why? Because human beings are unpredictable.
This is even more noticeable when we’re testing the UI with automated frameworks such as Selenium, Appium, and other front-end solutions for automated testing.
Avoid coverage for the sake of coverage
Even if the intention of trying to automate everything is noble and positive, sometimes that can lead us to write impossibly complex code that will be hard to maintain, fragile, and not cost-effective. To test unpredictable scenarios, we should use unpredictable methods (humans).
Where does the automation come in handy? With things like Regression Tests, Smoke Tests, and Performance Tests. The type of testing that always follows the same path and tends to be repetitive. Having that QA effort automated allows the human testers to focus on the fun stuff, which also keeps their concentration up and makes them more productive.
A good balance between Manual and Automated Tests is key in every QA Plan
How many manual tests vs. automated tests you have are up to you. That depends a lot on the type of system that’s being tested. There are systems that allow for bigger automated coverage than others. Sometimes if something seems too hard to automate, it’s because it shouldn’t be automated at all.
When our goal is to have 100% coverage of automated tests in our system, we’re very likely to run into these problems:
- Code that’s hard to maintain
- A Solution that seems to “have no end”
- Limitations of what our tools can do
- Slow progress (if any)
So, does that mean that we should focus only on covering a small portion of our system? Absolutely no. Not focusing on 100% coverage doesn’t mean that we should just cover the basics. What it means is that we should focus on adding tests that add value to our solution.
How can we decide what to automate, then?
There’s no golden rule for this, most of it falls into common sense. But usually, the best candidates for automation are:
- Smoke Tests: If your smoke tests are automated you will always have a quick way of knowing if your app is stable enough to proceed with the rest of the tests.
- Regression tests: A Regression Test Suite can get huge (and boring) really quickly. Automating this part of your tests will allow you to focus your QA Resources on testing new features with more concentration and less fatigue.
What not to expect from automated tests?
We should never expect to test new features with our automation. The first time a new feature is made is preferable to have actual eyes looking at it and testing it from as many sides as possible. Automated Tests are good for validating if something is still working as expected despite the changes made to the app. Not the other way around.
It’s a lot easier and more productive to write an automated test for a system that’s stable at the moment. A new feature cannot guarantee this. In an ideal world, a new feature is tested and documented by a human, and then, when that feature is tested as part of the regression tests it is automated based on the findings of the manual tests.
In conclusion, keep it simple, keep it tidy, and keep it meaningful.
This article was written by Mike Arias
Senior Software Testing Engineer of TechAID.
What is Context-Driven Testing?By Mike Arias
XPATH: With Great Power Comes Great ResponsibilityBy Mike Arias
OTHER POSTS YOU MIGHT LIKE
World Quality Report 2022-2023 Findings and Key Takeaways
World Quality Report 2022-23 Are you curious about the latest developments in the world of quality assurance? With the World Quality Report 2022-23, you can get an in-depth look at the current state of quality assurance, from industry trends to emerging technologies. Read on to…
Penetration Testing for Fintech. How to avoid cyber-attacks?
Penetration Testing Technology has been expanding for many years. Nowadays, it is an excellent tool for daily life that makes information extensive and easy to digest. However, it can also make our information public without notice. We use many mobile and financial applications daily, which…