
Hybrid Tests: Blurring the lines of the Automation Pyramid
Mike Cohn’s Test Automation Pyramid is a guide that is frequently cited as a good practice to follow. It suggests that:
- the bulk of your automated tests should be unit tests
- the next largest group of tests should be developed to target the service layer which includes the web services and business layer of your application
- there should be a very limited amount of UI tests, as they are more brittle and take the longest to execute
While it seems that just about everyone agrees with this in theory, I don’t see a lot of teams following this guide in practice. When there’s a new UI feature to test, we’re naturally inclined to automate those scenarios at the UI layer.
I’m with you! However, know that you don’t have to live in just one section of the Automation Pyramid. Just because you feel the need to test the feature at the UI layer doesn’t mean that every single step of the scenario has to be done there as well.
Let’s look at a few features and I’ll illustrate how to blur the lines within the pyramid. I’ll be using the ToolsQA site to demonstrate.
Feature 1: Search Catalog
As a user, I want to search the catalog so that I can find specific products
For this feature, the scenarios might include lots of different ways to search for a product. While this could certainly be done at the service layer, I’d feel a lot more comfortable automating this at the UI layer to ensure that the display of the search results (or error messages) is accurate. This is especially true for more advanced search result grids that allow for filtering, sorting, column organization, pagination, etc.
However, if you’re doing a ton of different search tests, it might make sense to do a few different variations on the UI, but write unit tests or utilize a Search service call for tests that are essentially exercising the same functionality but with different data.
Feature 2: Add Product to Cart
As a user, I want to view the details page of a product and add the product to my cart.
For this feature, the main test would be to verify that the user can add a product to the cart. The steps involved in the scenario would be:
- Search for the product
- Locate the product within the search results
- Click on the product
- Click the Add to Cart button
- Click Cart button and verify that the product is in the cart
In this scenario, you definitely want to make sure that the Add to Cart button works. Because of this, most people would concede to the fact that this scenario has to be automated on the UI. But if you evaluate the steps carefully, you’ll realize that step 4 is really where the testing begins. Everything else can be done outside of the UI.
From the work you’ve done on Feature 1, you already have automated scenarios for searching for a product, so it’s really tempting to reuse that framework for this test. Don’t do it! For one, it’ll be slower to execute this on the UI when in actuality, you don’t even have to. And more importantly, it’s not relevant to the test.
Let’s say there’s a regression with the Search feature and searching is now broken. If this Add to Cart scenario included Search, this test would now fail at the very first step, which means you’re losing the ability to use this test to verify what it’s actually intended to test: adding something to the cart.
Now, for this particular scenario, I’d still use the UI but I would utilize shortcuts. If I click on any product, I notice in the browser’s address bar that there’s a unique URL for each product. In this particular demo application, the unique identifier is the category + name. Your application might list the product’s ID. Knowing this, you can use your browser automation tool to go directly to your URL as opposed to navigating your way there through several steps.
We’ve now eliminated steps 1-3 and replaced it with a simple “Go to” and have eliminated the dependency on the Search feature. An additional benefit is that the test will run much faster and be less brittle.
- Go to the product’s URL
- Click the Add to Cart button
- Click Cart button and verify that the product is in the cart
Feature 3: Remove Product from Cart
As a user, I want to remove an item from my cart.
For this feature, the action happens in the cart. Yet, there are several steps that must occur before this can be tested.
- Search for the product
- Locate the product within the search results
- Click on the product
- Click the Add to Cart button
- Click Cart button
- Click the remove button next to the product and verify that the product is removed
The last step is the only thing that needs to be verified via the UI. We’ve already seen above how to eliminate steps 1-3. For step 4, you have already verified adding products to carts via the UI, so no need to do that yet again in this test. Instead, use a web service to place the item in the cart. The easiest approach would be to ask your developers if such a service exists. However, you can also find this on your own by inspecting the Add to Cart button in your browser’s dev tools.
The button will be an input element within a form. That form has everything you need to craft your service call. The action and method attributes on the form, along with the input name/value pairs are what to look for.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
<form class="product_form_ajax" enctype="multipart/form-data" action="http://store.demoqa.com/products-page/product-category/accessories/magic-mouse/" method="post" name="1"> <div class="wpsc_product_price"> <p class="pricedisplay specialprice 40"><span class="oldprice old_product_price_40">$200.00</span></p> <p class="pricedisplay normalprice 40"><span class="currentprice pricedisplay product_price_40">$150.00</span></p> <!-- multi currency code --> <span class="wpscsmall pricefloatright pricedisplay">IN: <span class="pricedisplay">$12,000.00</span></span><br> </div><!--close wpsc_product_price--> <input value="add_to_cart" name="wpsc_ajax_action" type="hidden"> <input value="40" name="product_id" type="hidden"> <div class="wpsc_buy_button_container group"> <div class="input-button-buy"><span><input value="Add To Cart" name="Buy" class="wpsc_buy_button" type="submit"></span> <div class="alert error"><p>Please select product options before adding to cart</p><span> </span></div> <div class="alert addtocart"><p>Item has been added to your cart!</p><span> </span></div> </div><!--close input-button-buy--> <div class="wpsc_loading_animation"> <img title="Loading" alt="Loading" src="http://store.demoqa.com/wp-content/themes/mio/images/ajax-loader.gif"> </div><!--close wpsc_loading_animation--> </div><!--close wpsc_buy_button_container--> </form> |
Given this information, we can craft a POST service call that we can use as a “go to” URL after opening the application.
1 |
http://store.demoqa.com/products-page/product-category/accessories/magic-mouse?wpsc_ajax_action=add_to_cart&product_id=40 |
This will add the item to the cart, which eliminates the need for steps 1-4. We can also replace step 5 if we’d like to. Instead of looking for the cart button and clicking it, we can simply go to the cart’s URL. Again, faster and less brittle!
- Add product to cart via web service call
- Go to Cart url
- Click the remove button next to the product and verify that the product is removed
If you’re worried about your UI not getting enough coverage, don’t. Notice that in all of the shortcuts in this demo, this same functionality was already exercised by some other automated scenario. So there was really no benefit to doing it again in another scenario. In your own projects, if you don’t have UI coverage for something that you think is important to cover on the UI, then go ahead and add it in, but once is good enough. For all other scenarios around that same area, utilize your application’s shortcuts. Your execution times and flaky test police will thank you for it.
Pingback: Five Blogs – 8 February 2017 – 5blogs
Diogo Nunes
This is an insightful post. I’m currently not doing this, and my UI tests are slow, because for every test I have to start a new WebDriver, login, and navigate to the page under test. I want to keep separate WebDriver instances, but that “shortcuts” idea really rings a bell.
I thought I should avoid shortcuts and just use the UI, like a real user, but I get your point, that was already tested on some other check, so it’s just adding more time overhead. Cheers!
Pingback: Testing Bits – 2/5/17 – 2/11/17 | Testing Curator Blog
Xavier
Hi,
Nice article 🙂 However, when you arrive at this last scenario “Add product to cart via web service call, Go to Cart url…”, I think that the scenario is becoming too technical for the Business People to understand. How do you manage to keep them on board ? Do you have any insight on this ?
Thanks !
Xavier
Angie Jones
Hi Xavier. Usually, the business folks are ok with me giving them the high-level scenario and aren’t interested in the detailed steps I will take.
Andy Carrington-Chappell
Nice article, and a great primer to start thinking logically about testing to minimise UI interaction. After reading this and a recent tweet by the Friendly Tester I am attempting to find the ‘seams’ in our systems to improve how we automate our own testing.
Derick
Interesting Article.
Bala Kopparthi
Excellent Article…Very True
Tony Briceño
Interesting article,
Nadeesha
Very useful article. Thank You
Martin Hernandez
Thank you for this important information!
Konrad
hi, the website used for this demo does not seem to be achievable…
Ananth Ganapathiraman
Hi Anjie,
This is the only approach I have advocated for manual testing :). If a functional test exercises the ‘Search’ functionality separately, then for any other test where this forms as a navigation step, the QA team should only use shortcuts (wherever possible). This is to ensure that they do not block 100 scenarios in case of a fault in the Search functionality. But this approach is difficult to sell in teams that are old-fashioned or stuck in their ways or are metrics-driven or want to monetize waiting periods for consultants etc.
I am so glad this article is even published albeit for automated testing. I will quote this article in all my future projects.
Thank You
Jesse Smith
Thank you
Akankssha
Really helpful thanks for creating this page.
Vivek
Hey Angie, Great post. l love it.
Emre
It was so insightful and eye-opened. Thank you.