Hybrid Tests: Blurring the lines of the Automation Pyramid

Mike Cohn’s Test Automation Pyramid is a guide that is frequently cited as a good practice to follow. It suggests that:

• the bulk of your automated tests should be unit tests
• the next largest group of tests should be developed to target the service layer which includes the web services and business layer of your application
• there should be a very limited amount of UI tests, as they are more brittle and take the longest to execute

While it seems that just about everyone agrees with this in theory, I don’t see a lot of teams following this guide in practice. When there’s a new UI feature to test, we’re naturally inclined to automate those scenarios at the UI layer.

I’m with you! However, know that you don’t have to live in just one section of the Automation Pyramid. Just because you feel the need to test the feature at the UI layer doesn’t mean that every single step of the scenario has to be done there as well.

Let’s look at a few features and I’ll illustrate how to blur the lines within the pyramid. I’ll be using the ToolsQA site to demonstrate.

Feature 1: Search Catalog
As a user, I want to search the catalog so that I can find specific products

For this feature, the scenarios might include lots of different ways to search for a product. While this could certainly be done at the service layer, I’d feel a lot more comfortable automating this at the UI layer to ensure that the display of the search results (or error messages) is accurate. This is especially true for more advanced search result grids that allow for filtering, sorting, column organization, pagination, etc.

However, if you’re doing a ton of different search tests, it might make sense to do a few different variations on the UI, but write unit tests or utilize a Search service call for tests that are essentially exercising the same functionality but with different data.

Feature 2: Add Product to Cart
As a user, I want to view the details page of a product and add the product to my cart.

For this feature, the main test would be to verify that the user can add a product to the cart. The steps involved in the scenario would be:

In this scenario, you definitely want to make sure that the Add to Cart button works. Because of this, most people would concede to the fact that this scenario has to be automated on the UI. But if you evaluate the steps carefully, you’ll realize that step 4 is really where the testing begins. Everything else can be done outside of the UI.

From the work you’ve done on Feature 1, you already have automated scenarios for searching for a product, so it’s really tempting to reuse that framework for this test. Don’t do it! For one, it’ll be slower to execute this on the UI when in actuality, you don’t even have to. And more importantly, it’s not relevant to the test.

Let’s say there’s a regression with the Search feature and searching is now broken. If this Add to Cart scenario included Search, this test would now fail at the very first step, which means you’re losing the ability to use this test to verify what it’s actually intended to test: adding something to the cart.

Now, for this particular scenario, I’d still use the UI but I would utilize shortcuts. If I click on any product, I notice in the browser’s address bar that there’s a unique URL for each product. In this particular demo application, the unique identifier is the category + name. Your application might list the product’s ID. Knowing this, you can use your browser automation tool to go directly to your URL as opposed to navigating your way there through several steps.

We’ve now eliminated steps 1-3 and replaced it with a simple “Go to” and have eliminated the dependency on the Search feature. An additional benefit is that the test will run much faster and be less brittle.

Feature 3: Remove Product from Cart
As a user, I want to remove an item from my cart.

For this feature, the action happens in the cart. Yet, there are several steps that must occur before this can be tested.

The last step is the only thing that needs to be verified via the UI. We’ve already seen above how to eliminate steps 1-3. For step 4, you have already verified adding products to carts via the UI, so no need to do that yet again in this test. Instead, use a web service to place the item in the cart. The easiest approach would be to ask your developers if such a service exists. However, you can also find this on your own by inspecting the Add to Cart button in your browser’s dev tools.

The button will be an input element within a form. That form has everything you need to craft your service call. The action and method attributes on the form, along with the input name/value pairs are what to look for.

Given this information, we can craft a POST service call that we can use as a “go to” URL after opening the application.

This will add the item to the cart, which eliminates the need for steps 1-4. We can also replace step 5 if we’d like to. Instead of looking for the cart button and clicking it, we can simply go to the cart’s URL. Again, faster and less brittle!

If you’re worried about your UI not getting enough coverage, don’t. Notice that in all of the shortcuts in this demo, this same functionality was already exercised by some other automated scenario. So there was really no benefit to doing it again in another scenario. In your own projects, if you don’t have UI coverage for something that you think is important to cover on the UI, then go ahead and add it in, but once is good enough. For all other scenarios around that same area, utilize your application’s shortcuts. Your execution times and flaky test police will thank you for it.