Verifying Entire API Responses

Most people do a pretty lazy job of testing APIs. In my ABCs of APIs workshop, I ask people to manually test an API. When looking at a response, they typically glance at it, and maybe carefully review some of the key fields. Same goes for automated tests – only the key fields are typically asserted against.

Let’s look at this GET call:

This returns the following response:

Given this response, many unit tests will make sure the response code is 200 and that the body is not null. Functional tests may go a bit further and verify some of the key fields of the body.

Most people won’t script a test that verifies EVERY bit of this response, mostly because it’s really tedious to do and arguably some of the fields may pose a lower risk if they are incorrect.

My argument is that you don’t know how every customer will use your API, and therefore can’t be sure which of these fields are most important to them.

Because of this, I’ve been seeking a way to verify an entire response with a single assertion. I’d gotten close to being able to do this by deserializing the API response into a POJO and comparing the resulting object with an expected object. But even with this approach, I needed to code up the POJO, and build the expected object in my code. Still a bit tedious.

Luckily, Mark Winteringham taught a workshop on Approval Tests (created by Llewellyn Falco) and showed us how to verify an entire response body! I went home and played with it more on my own and am loving it!

After the first run of your test, Approval Tests will save your expected result to a file. Then on each subsequent run, it will compare the new result to what’s saved as the expected result. If the results differ, the test fails and a diffing program such DiffMerge can show you the differences between the files.

 

 

Dependencies

I created a pom file with Approval Tests, TestNG (as the test runner), and Rest-Assured (as the API executor). Note that Approval Tests can work alongside any other test runner and API executor, and there’s support for several programming languages as well.

 

Testing Response Body

In this test, I want to verify the entire body of the response. I can do that by calling Approvals.verify and passing in the body. While this verifies the body, the status code could still be wrong. So, I add one more call to Rest-Assured’s statusCode method to verify that as well.

When executed the first time, this will fail because there’s no approved file saved yet.

I take the result, make sure that it’s ok, and then save this into the approved file. This becomes my golden master.

When I run this again, Approval Test compares the received file (which now contains the new info) with the approved file. Since they both match, the test passes.

 

Testing Entire Response

While testing the response body along with the status code may be enough, it doesn’t hurt to go ahead and test the entire response – including the headers. I’ve certainly found bugs here before, such as with headers that contain pagination and such.

I created a new test to verify the entire response. Since the response as a whole also includes the status code, I no longer need a separate assertion just for that.

 

Dynamic Data

You’ll quickly run into an issue with using this technique: most responses are not static. For POST calls, the response may include a newly generated ID. Headers may include the date or cookie information that changes every time the test is run. Fortunately, I remembered Mark teaching us a little trick to deal with this.

Since we’re working with Strings here, we can simply do a replaceAll and use a regular expression to find the lines that we know are dynamic and then mask those lines.

In my response, there were three dynamic headers, so I masked them as follows:

The file contents are then saved as:

 

Download Code

I’ve cleaned this up a bit and placed it on Github for your convenience.

Get Demo Code