Top

Hybrid Tests: Blurring the lines of the Automation Pyramid

Hybrid Tests: Blurring the lines of the Automation Pyramid

Mike Cohn’s Test Automation Pyramid is a guide that is frequently cited as a good practice to follow. It suggests that:

  • the bulk of your automated tests should be unit tests
  • the next largest group of tests should be developed to target the service layer which includes the web services and business layer of your application
  • there should be a very limited amount of UI tests, as they are more brittle and take the longest to execute

While it seems that just about everyone agrees with this in theory, I don’t see a lot of teams following this guide in practice. When there’s a new UI feature to test, we’re naturally inclined to automate those scenarios at the UI layer.

I’m with you! However, know that you don’t have to live in just one section of the Automation Pyramid. Just because you feel the need to test the feature at the UI layer doesn’t mean that every single step of the scenario has to be done there as well.

Let’s look at a few features and I’ll illustrate how to blur the lines within the pyramid. I’ll be using the ToolsQA site to demonstrate.

    

Feature 1: Search Catalog
As a user, I want to search the catalog so that I can find specific products

For this feature, the scenarios might include lots of different ways to search for a product. While this could certainly be done at the service layer, I’d feel a lot more comfortable automating this at the UI layer to ensure that the display of the search results (or error messages) is accurate. This is especially true for more advanced search result grids that allow for filtering, sorting, column organization, pagination, etc.

However, if you’re doing a ton of different search tests, it might make sense to do a few different variations on the UI, but write unit tests or utilize a Search service call for tests that are essentially exercising the same functionality but with different data.

    

Feature 2: Add Product to Cart
As a user, I want to view the details page of a product and add the product to my cart.

For this feature, the main test would be to verify that the user can add a product to the cart. The steps involved in the scenario would be:

  1. Search for the product
  2. Locate the product within the search results
  3. Click on the product
  4. Click the Add to Cart button
  5. Click Cart button and verify that the product is in the cart

In this scenario, you definitely want to make sure that the Add to Cart button works. Because of this, most people would concede to the fact that this scenario has to be automated on the UI. But if you evaluate the steps carefully, you’ll realize that step 4 is really where the testing begins. Everything else can be done outside of the UI.

From the work you’ve done on Feature 1, you already have automated scenarios for searching for a product, so it’s really tempting to reuse that framework for this test. Don’t do it! For one, it’ll be slower to execute this on the UI when in actuality, you don’t even have to. And more importantly, it’s not relevant to the test.

Let’s say there’s a regression with the Search feature and searching is now broken. If this Add to Cart scenario included Search, this test would now fail at the very first step, which means you’re losing the ability to use this test to verify what it’s actually intended to test: adding something to the cart.

Now, for this particular scenario, I’d still use the UI but I would utilize shortcuts. If I click on any product, I notice in the browser’s address bar that there’s a unique URL for each product. In this particular demo application, the unique identifier is the category + name. Your application might list the product’s ID. Knowing this, you can use your browser automation tool to go directly to your URL as opposed to navigating your way there through several steps.

We’ve now eliminated steps 1-3 and replaced it with a simple “Go to” and have eliminated the dependency on the Search feature. An additional benefit is that the test will run much faster and be less brittle.

  1. Go to the product’s URL
  2. Click the Add to Cart button
  3. Click Cart button and verify that the product is in the cart

    

Feature 3: Remove Product from Cart
As a user, I want to remove an item from my cart.

For this feature, the action happens in the cart. Yet, there are several steps that must occur before this can be tested.

  1. Search for the product
  2. Locate the product within the search results
  3. Click on the product
  4. Click the Add to Cart button
  5. Click Cart button
  6. Click the remove button next to the product and verify that the product is removed

The last step is the only thing that needs to be verified via the UI. We’ve already seen above how to eliminate steps 1-3. For step 4, you have already verified adding products to carts via the UI, so no need to do that yet again in this test. Instead, use a web service to place the item in the cart. The easiest approach would be to ask your developers if such a service exists. However, you can also find this on your own by inspecting the Add to Cart button in your browser’s dev tools.

The button will be an input element within a form. That form has everything you need to craft your service call. The action and method attributes on the form, along with the input name/value pairs are what to look for.

Given this information, we can craft a POST service call that we can use as a “go to” URL after opening the application.

This will add the item to the cart, which eliminates the need for steps 1-4. We can also replace step 5 if we’d like to. Instead of looking for the cart button and clicking it, we can simply go to the cart’s URL. Again, faster and less brittle!

  1. Add product to cart via web service call
  2. Go to Cart url
  3. Click the remove button next to the product and verify that the product is removed

    

If you’re worried about your UI not getting enough coverage, don’t. Notice that in all of the shortcuts in this demo, this same functionality was already exercised by some other automated scenario. So there was really no benefit to doing it again in another scenario. In your own projects, if you don’t have UI coverage for something that you think is important to cover on the UI, then go ahead and add it in, but once is good enough. For all other scenarios around that same area, utilize your application’s shortcuts. Your execution times and flaky test police will thank you for it.

Angie Jones
17 Comments
  • Diogo Nunes

    This is an insightful post. I’m currently not doing this, and my UI tests are slow, because for every test I have to start a new WebDriver, login, and navigate to the page under test. I want to keep separate WebDriver instances, but that “shortcuts” idea really rings a bell.
    I thought I should avoid shortcuts and just use the UI, like a real user, but I get your point, that was already tested on some other check, so it’s just adding more time overhead. Cheers!

    February 8, 2017 at 5:58 am Reply
  • Xavier

    Hi,
    Nice article 🙂 However, when you arrive at this last scenario “Add product to cart via web service call, Go to Cart url…”, I think that the scenario is becoming too technical for the Business People to understand. How do you manage to keep them on board ? Do you have any insight on this ?
    Thanks !
    Xavier

    November 6, 2017 at 10:58 am Reply
    • Angie Jones

      Hi Xavier. Usually, the business folks are ok with me giving them the high-level scenario and aren’t interested in the detailed steps I will take.

      November 6, 2017 at 12:02 pm Reply
  • Andy Carrington-Chappell

    Nice article, and a great primer to start thinking logically about testing to minimise UI interaction. After reading this and a recent tweet by the Friendly Tester I am attempting to find the ‘seams’ in our systems to improve how we automate our own testing.

    May 9, 2018 at 2:14 am Reply
  • Derick

    Interesting Article.

    January 9, 2019 at 2:10 am Reply
  • Bala Kopparthi

    Excellent Article…Very True

    December 4, 2019 at 4:53 pm Reply
  • Tony Briceño

    Interesting article,

    With this approach I can see many advantages for my future test steps: setting up the preconditions by skipping the “step-by-step” and going straight to the point
    could save execution time and even maintenance for the preconditions steps.
    So, would you recommend for background scenario steps to be like this way? since usually background steps are a copy of an existent coverage test.
    Thanks ,
    April 21, 2020 at 12:23 pm Reply
  • Nadeesha

    Very useful article. Thank You

    June 19, 2020 at 11:34 am Reply
  • Martin Hernandez

    Thank you for this important information!

    January 20, 2021 at 8:32 pm Reply
  • Konrad

    hi, the website used for this demo does not seem to be achievable…

    May 11, 2021 at 1:11 am Reply
  • Ananth Ganapathiraman

    Hi Anjie,

    This is the only approach I have advocated for manual testing :). If a functional test exercises the ‘Search’ functionality separately, then for any other test where this forms as a navigation step, the QA team should only use shortcuts (wherever possible). This is to ensure that they do not block 100 scenarios in case of a fault in the Search functionality. But this approach is difficult to sell in teams that are old-fashioned or stuck in their ways or are metrics-driven or want to monetize waiting periods for consultants etc.

    I am so glad this article is even published albeit for automated testing. I will quote this article in all my future projects.

    Thank You

    February 18, 2022 at 1:52 pm Reply
  • Jesse Smith

    Thank you

    February 21, 2022 at 6:29 pm Reply
  • Akankssha

    Really helpful thanks for creating this page.

    March 28, 2022 at 12:10 am Reply
  • Vivek

    Hey Angie, Great post. l love it.

    May 24, 2022 at 11:31 am Reply
  • Emre

    It was so insightful and eye-opened. Thank you.

    January 13, 2023 at 1:03 pm Reply

Post a Comment