Top

Verifying Entire API Responses

Verifying Entire API Responses

Most people do a pretty lazy job of testing APIs. In my ABCs of APIs workshop, I ask people to manually test an API. When looking at a response, they typically glance at it, and maybe carefully review some of the key fields. Same goes for automated tests – only the key fields are typically asserted against.

Let’s look at this GET call:

This returns the following response:

Given this response, many unit tests will make sure the response code is 200 and that the body is not null. Functional tests may go a bit further and verify some of the key fields of the body.

Most people won’t script a test that verifies EVERY bit of this response, mostly because it’s really tedious to do and arguably some of the fields may pose a lower risk if they are incorrect.

My argument is that you don’t know how every customer will use your API, and therefore can’t be sure which of these fields are most important to them.

Because of this, I’ve been seeking a way to verify an entire response with a single assertion. I’d gotten close to being able to do this by deserializing the API response into a POJO and comparing the resulting object with an expected object. But even with this approach, I needed to code up the POJO, and build the expected object in my code. Still a bit tedious.

Luckily, Mark Winteringham taught a workshop on Approval Tests (created by Llewellyn Falco) and showed us how to verify an entire response body! I went home and played with it more on my own and am loving it!

After the first run of your test, Approval Tests will save your expected result to a file. Then on each subsequent run, it will compare the new result to what’s saved as the expected result. If the results differ, the test fails and a diffing program such DiffMerge can show you the differences between the files.

 


Video by Clare Macrae

 

Dependencies

I created a pom file with Approval Tests, TestNG (as the test runner), and Rest-Assured (as the API executor). Note that Approval Tests can work alongside any other test runner and API executor, and there’s support for several programming languages as well.

 

Testing Response Body

In this test, I want to verify the entire body of the response. I can do that by calling Approvals.verify and passing in the body. While this verifies the body, the status code could still be wrong. So, I add one more call to Rest-Assured’s statusCode method to verify that as well.

When executed the first time, this will fail because there’s no approved file saved yet.

I take the result, make sure that it’s ok, and then save this into the approved file. This becomes my golden master.

When I run this again, Approval Test compares the received file (which now contains the new info) with the approved file. Since they both match, the test passes.

 

Testing Entire Response

While testing the response body along with the status code may be enough, it doesn’t hurt to go ahead and test the entire response – including the headers. I’ve certainly found bugs here before, such as with headers that contain pagination and such.

I created a new test to verify the entire response. Since the response as a whole also includes the status code, I no longer need a separate assertion just for that.

 

Dynamic Data

You’ll quickly run into an issue with using this technique: most responses are not static. For POST calls, the response may include a newly generated ID. Headers may include the date or cookie information that changes every time the test is run. Fortunately, I remembered Mark teaching us a little trick to deal with this.

Since we’re working with Strings here, we can simply do a replaceAll and use a regular expression to find the lines that we know are dynamic and then mask those lines.

In my response, there were three dynamic headers, so I masked them as follows:

The file contents are then saved as:

 

Download Code

I’ve cleaned this up a bit and placed it on Github for your convenience.

Get Demo Code

Angie Jones
9 Comments
  • David Garratt

    Really cool

    Feels like this could be expanded on to create something like Applitools Eyes for Apis (it rhymes!)
    Run tests
    Put the baseline responses into a DB
    Human review of initial responses to set the baseline
    Allow for optional/changing values through annotation of some sort
    Run tests
    Flag differences between baseline and actual for human review
    David Garratt
    June 24, 2019 at 10:21 am Reply
  • Nibs

    Thanks Angie, But for an API responses with lot of dynamic data, seems to be bit difficult. JSON schema validations should be right fit there.

    June 25, 2019 at 12:09 am Reply
  • Shawn Bacot

    This is a really interesting approach. I’m interested to hear what you think about using a schema validator like AJV for Javascript to validate responses. It allows you to check for the present of required properties, value types in each response and provide expected data values to validate against as well. I’ve been doing that coupled with focused assertions for expected response values, particularly when creating and updating objects with some success.

    June 25, 2019 at 11:49 am Reply
  • Lakshmikanthan

    This is really interesting.

    I have a use case where i am trying to see if this solution best fits
    My test class has single test method which calls 3 API’s ( 3 different scenarios with 3 different json output). I need to run this test method against different test data ( in excel)which i am passing to the test using the data provider
    How efficiently this can be achieved using this Approval tests? Thanks in advance 🙂
    July 24, 2019 at 5:55 am Reply
  • Will O.

    This is great. I’ve tried it out and it works as expected. However, I need some help figuring out how I can move the approved.txt file(s) to a separate location for better structure without breaking the link to the test 🙂

    July 26, 2019 at 3:34 am Reply
  • Vindhiyan
    Thanks for the article

    You had masked headers in the test.
    How do you mask dynamic values in the response body?
    August 18, 2019 at 5:31 pm Reply

Post a Comment