Test Automation for PDF Files

For years, the automated verification of PDFs was incredibly challenging, if not impossible. Because of this, teams would automate their UI tests but would skip the part where they verify that their PDF artifacts were accurate. This then became the boring, mundane, error-prone task left for the testers to repeat release after release.

Since then, visual validation tools such as Applitools Eyes have hit the scene making the automated regression testing of the look and feel of an application possible. A common question I receive is “does Applitools work for PDFs?”. I knew the answer was yes, but I decided to actually give it a try myself to see exactly how it works.

There’s actually two ways to invoke PDF test automation. The one described in the tool’s tutorial page shows how to execute the PDF validation via the command line. However, as an automation engineer, I wondered if it was possible to do this via my existing automation framework. Since the magic happens via a Java CLI command, I was pretty sure it should also work from my code. But I wanted to try it out just to be sure. It worked!

I’ll detail both approaches.

Command Line Interface (CLI)

Applitools provides an executable, ImageTester.jar, which is a tool that verifies stand-alone images and also PDF files.

It’s pretty straightforward to use. You put your PDF files in a directory and run a command from your terminal to have this tool verify all files within that directly. Alternatively, you can specify an individual PDF filename and it will only verify that particular file.

The -k argument is your API key which you can obtain by opening a free account. And the -f  argument is the path to the directory or file that you want verified.

I moved the ImageTester.jar into a directory and also added another directory there called Invoice_PDFs where I stored this PDF file. I then ran the command and voila, the test was executed!

 

The first time I ran this, a baseline was saved, and then every time this was run again, the PDF was compared against the baseline. If anything changed on the PDFs, we’d get an error message on the console and link to review the differences in the Applitools dashboard.

 

Code

The CLI approach is cool, but I got to thinking about how I would want to use this as an automation engineer. It would be in the midst of an automated scenario where I’ve taken action on the UI, am downloading the resulting PDF, and now want to verify it.

So, I wrote an automated test for Invoice Simple that uses the UI to create a new invoice, then downloads a PDF of that invoice and then uses the ImageTester to verify the PDF.

After writing all the UI code, I needed to add the following as well:

1. Code to move the PDF file from my computer’s default download directory to the directory I store the PDF files that I want verified.
   
   Move file to test directory

   Java

   File downloadedPDF = new File("/Users/angie/Downloads/" + invoiceNumber + ".pdf"); String destination = "resources/Invoice_PDFs/" + invoiceNumber + ".pdf"; FileUtils.moveFile(downloadedPDF, destination);

   1 2 3

   File downloadedPDF = new File("/Users/angie/Downloads/" + invoiceNumber + ".pdf"); String destination = "resources/Invoice_PDFs/" + invoiceNumber + ".pdf"; FileUtils.moveFile(downloadedPDF, destination);

   FileUtils.moveFile

   Java

   public static boolean moveFile(File file, String destination){ File existingFile = new File(destination); if(existingFile.exists()){ existingFile.delete(); } return file.renameTo(new File(destination)); }

   1 2 3 4 5 6 7 8 9

   public static boolean moveFile(File file, String destination){           File existingFile = new File(destination);         if(existingFile.exists()){             existingFile.delete();         }           return file.renameTo(new File(destination));     }

   
   
   
2. Execute the ImageTester.jar command. I wrote a utility method so that I could reuse it from any test.
   
   Utility method to execute PDF validation

   Java

   public static boolean validatePDF(String filepath) throws IOException, InterruptedException { String command = String.format( "java -jar resources/ImageTester.jar -k %s -f %s", System.getProperty("applitools.api.key"), filepath); Process process = Runtime.getRuntime().exec(command); process.waitFor(); String stream = IOUtils.toString(process.getInputStream(), "UTF-8"); System.out.println(stream); if(stream != null && stream.contains("Mismatch")){ return false; } return true; }

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

   public static boolean validatePDF(String filepath) throws IOException, InterruptedException {         String command = String.format(                 "java -jar resources/ImageTester.jar -k %s -f %s",                 System.getProperty("applitools.api.key"),                 filepath);           Process process = Runtime.getRuntime().exec(command);         process.waitFor();         String stream = IOUtils.toString(process.getInputStream(), "UTF-8");         System.out.println(stream);           if(stream != null && stream.contains("Mismatch")){             return false;         }           return true;     }

   Then from my test, I call this method and assert on the result.
   Assertion within test method

   Java

   Assert.assertTrue("Error validating PDF", validatePDF(destination));

   1	Assert.assertTrue("Error validating PDF", validatePDF(destination));

   
   
   
3. There is a date on the PDF file indicating when this file was generated. Well, that date will change each day that this test runs. Fortunately, Applitools has a way to ignore certain regions of the PDF file. After my initial test ran and the baseline was captured, I was able to go to the dashboard and specify to ignore the date area.
   
   
   
   I really like how flexible this tool is. There’s a host of other arguments you can use as well.

This all worked like a charm and was much simpler than I anticipated. You can find all of my code for this automated test for PDF files on my Github.

See Code on Github