My indifference-hate relationship with Gherkin

As an automation architect, I get a lot of questions and requests from various clients about how to implement Gherkin. Over the years, my opinion has gone from general ambivalence to outright dread. Few organizations understand the value that Gherkin provides and even fewer understand the costs. That’s not to say I think there is never a reason to leverage Gherkin. When used correctly it can add value to many projects.

The good

I like the idea behind Gherkin. Gherkin is a very effective way for a business to convey its wants and needs to development. It abstracts away a lot of the technical detail that the business doesn’t understand and/or care about. I have actually worked in an environment where BAs would write requirements using the Gherkin syntax and it really helped make sure the business was getting what they expected.

The bad

The business is often left out

Gherkin is meant to facilitate communication between the business and technical teams. If your business is never seeing or reviewing the Gherkin, you are missing out on the primary benefit. If the business is not involved, I would argue Gherkin is more overhead for little to no value.

It’s seen as a magic bullet

With Gherkin, you can have all your tests written before a single line of code is written. On top of that, you can just automate your test steps and now you don’t even need manual tests. It sounds like a great idea! In many ways it is, but any seasoned tester can tell you that this is only part of an effective testing strategy. First, these tests are only as good as the Gherkin and it’s not easy to write effective Gherkin. Secondly, Gherkin is typically heavily focused on basic happy path testing. Finally, no matter how good, verbose, or clever your Gherkin is, it will never be as effective as a skilled exploratory tester.

It’s used as a way to get manual testers to write automation

It is true that Gherkin tools can be used for writing automation, but it’s penny wise and pound foolish. When used this way, it makes the difficult job of building a robust automation framework considerably harder and the relatively simple task of writing tests slightly easier. On top of that it encourages the idea that manual and automated tests are equivalent, they are not.

The true cost

I see the use of the Gherkin tools to be where the lion’s share of the cost comes into play. The cost is not in the tools themselves, many of them are free or relatively inexpensive. The cost comes from what the tools do you your automation code. They cause more convoluted code that is harder to write and maintain.

Tools add an extra abstraction layer

Underneath the hood, scenarios are doing little more than calling public functions.

It takes work to make sure your steps don’t collide

Generic steps like “I click on the button” need to be avoided, because all of your steps are global. The person reading the steps will know, based on the context of the test, what button they are supposed to click. Without some pretty ugly code, your coded steps have no idea of this context.

Scenario context is basically a global variable dumping ground

If I want to use something from a previous step, I need to save it in the scenario context. Now my steps need to depend on previous steps placing required context and objects into this global catchall.

Step failures can be misleading

When a step fails, there is often an assumption made as to what actually happened. Steps often, by necessity, need to do more than exactly what the step says. This can result in test steps failing for many different reasons. Those reviewing the results often conflate errors within a step, thus failing to recognize new or unrelated failures.

Gherkin tools make dealing with complex data objects and states difficult

By design, Gherkin is meant to be very easily read by humans. One way of doing this is to only allow primitive types (numbers and strings) to be passed around, which means you need to maintain the more complex types in your scenario context. This also means each step needs to be very careful about how it uses and alters these complex types as they are stored in the global dumping ground of scenario context.

The code ends up being very procedural

In general, most modern developers avoid procedural programming. It tends to be much harder to maintain and extend. This is exactly why your test automation should avoid procedural programming as well. The biggest challenge for test automation is not that it takes time to build it, it’s how much it costs to maintain it and procedural programming can significantly raise this cost.

Leveraging Gherkin without hurting your project

Here are the three keys I use on my projects to effectively leverage Gherkin:

• Make sure the business is involved.
• Understand that it is an important part of testing, but it is not an effective testing strategy by itself.
• Keep the spirit but dump the tools.

Go from

Feature: Login tests In order to verify logins Users should only be able to login with valid credentials

@Login Scenario: Login with valid credentials Given I open the login page And enter the username Ted and password 123 When I press the login button Then the homepage is displayed

@Login Scenario: Login blocked when using invalid credentials Given I open the login page And enter the username NAME and password BAD_PASS When I press the login button Then the login page displays a login error

to