10 signs your automation is in trouble

Let’s face it, automation isn’t cheap. However, when done well, it can make your testing effort far more efficient and help you get to market faster. When done poorly, it can burn through your budget without providing any real benefit. In the following list, I will highlight some warning signs that indicate your automation effort is in trouble. Hopefully, this knowledge will help you find gaps and improve your current and future automation efforts.

1. Engineers can’t run their tests on demand

The first thing I do when I go into a new environment with existing automation is ask to see the automation run. It is shocking how often the engineers can’t execute their own tests. This situation almost always means that the code is a mess and they aren’t keeping up on maintenance, or that the code never worked in the first place.

2. Using record and playback

Record and playback is appropriate for some situations, but I have yet to see it be the right long-term tool for any project. Record and playback makes automation buildout easy, but it also requires a lot of maintenance. That maintenance cost almost always eats up any value you may have gained by using record and playback in the first place.

3. The automation technology stack is different from the development technology stack

This is not necessarily a problem, but it is a red flag. Remember: When your automation stack is different from your development stack, you are adding additional infrastructure cost. This also makes it difficult for development to support the automation effort as they are (sometimes literally) talking in different languages.

4. No code reviews

Code reviews are a critical part of any software development project, and automation is software development. Reviews have four primary benefits. First, they help eliminate single points of failure by having more than one person familiar with any single piece of code. Second, they help you find problems in the code earlier — and the earlier you find problems, the easier (and cheaper) it is to fix those problems. Third, it forces engineers to write cleaner code, which is especially good for long-term maintenance. Lastly, code reviews are good training tools for both the reviewer and reviewee.

5. No continuous integration

As I mentioned before, automation is development. There is no reason in a modern development environment not to have continuous integration set up. This allows engineers to locate coding errors quickly. The longer it takes your engineers to realize their code is broken, the longer it takes to fix.

6. Your automation engineers are just writing UI tests

There are three main problems with just writing just UI automation tests. The first problem is that they are the most expensive tests to build and maintain. Second, you are not adequately testing your service and database layers. Third, on top of that, you are almost certainly missing the real power of automation. Automation is very good at tedious/repetitive work, which I see manual testers doing all the time. Leveraging automation to do repetitive tasks opens up your manual testers to do more interesting and valuable testing. The most valuable automation I ever wrote tested nothing, but it saved my manual testers roughly four hours each deployment by automating some basic, but laborious, setup steps.

7. Bad reporting

The product of testing is information, and the communication of that information is dependent on the quality of the reports. If no one consumes that data, which is often the case, then there is no point generating it in the first place. There are two big reasons why automation results may not be consumed. The first is that the manual and automation tests report in separate systems, and after a relatively short time, people tend to just look at one system for results. This is why single source reporting is important. The second reason is that it’s no one’s job to review automated test results and it really needs to be. Your automation may be finding serious bugs, but, without a review process, this information is never propagated to the team.

8. Test data is non-deterministic

Running automation against a non-deterministic (or unknown) data set is tricky. That’s not to say that it can’t be done, but it can get tricky. A deterministic data set allows you to test against a source of truth, but non-deterministic sets have either no source or a calculated source of truth. This often results in either a high number of tests failing that should pass and/or tests passing that should fail. It can defeat the entire purpose of writing automation in the first place.

9. 100% pass rate

If all your tests pass all the time, you are not testing that much, development has ended and/or you have tests passing that should be failing. I have yet to see an automation suite of any reasonable size passing everything.

10. Testing is 100% automated

Automation is very good at certain types of testing; if you need, for instance, to do highly repetitive checking, then automation is ideal. If you want to test a new feature and care about look, feel, usability, etc. then automation is typically a poor choice. Remember that automation tests exactly what you tell it to — no more, no less. An effective testing effort ideally has a mix of manual and automated testing working in concert together.