Test automation for an API: Postman, Newman and Jenkins
What is test automation?
The test automation is to use a program to control the execution of tests and check if the results are what we were expecting. To explain how to automate tests on an API, three programs will be needed: Postman, Newman and Jenkins.
Postman
The first program that we are going to use to achieve test automation is Postman, which can be defined as follows:
"The main objective of Postman is to help build APIs quickly allowing the agile creation of requests and workflows through collections"
Environment
The first thing we have to do before starting to create tests is to prepare an environment for the server in which the tests will be run. To create an environment we go to Manage environments → Add. Then we can start adding key-value pairs corresponding to variables that we will use multiple times in requests.
As an example, I have chosen the following API to explain how it works: swapi.co/api , so we are going to save it in a variable that we will call host:
This API does not need the user to authenticate, but if it did, we could save the username and password in the environment.
Collection
Postman allows you to create collections to store requests, so the first thing we need is to create a collection, providing its name (required) and a description.
Then we can create a request, which will look something like this:
In this example we try to get a list of spaceships, so in the screenshot you can see:
-
Name of the request.
-
Method: It can be GET, POST, PUT, DELETE ...
-
API: The full address would be swapi.co/api/starships, but since we already save swapi.co/api in the host environment variable we just have to write {{host}} to use it.
-
Environment.
The second part of creating a request is the Tests section, designed to create asserts that verify that the information returned by the server is what we were expecting and values can also be assigned to new environment or global variables to use later. For example:
In this test we have an assert that verifies that the status code is 200 and also a loop to store information about a certain ship of all the returned ones in an environment variable. On the right side we have a series of pieces of code that help the user to create tests.
The next step would be to save the request by pressing the "Save" button and choosing where to save it. We have the option of saving it directly in the collection:
Or in a folder. Storing them in folders has the advantage of being able to isolate their execution from the rest of the requests.
Execution of tests
To execute the created request, you must first save it (Important!) And then send it by clicking on “Send”. We are interested in two parts of the answer: Body, where we see the json returned by the server with all the information and Tests, where the result of the asserts will be seen.
In this case one of the 2 tests failed because it was specified in the tests that it was expected to receive more than 100 ships and if we look at the information received there are fewer.
Once we have all the requests we want together with their tests, they can be run together. For that we need to open the "Collection Runner":
To run the tests we must follow the following steps:
-
Choose to run the entire collection or just a folder.
-
Choose the environment where we have the variables stored.
-
Number of times we want to run the tests.
-
Start Test
Once the execution is finished, the results can be seen on the right side of the screen. At the top, a summary will appear in which you can see how many tests failed, how many failed, and how long they took to run. The same will appear below, but for each request made.
Newman
Postman has a Command Line Version called Newman that has many utilities but we only need one: It allows the integration of collections created in Postman with a continuous integration system such as Jenkins , which is a key step to achieve test automation .
Installing Newman is very easy:
Before starting to run the test we must first export the environment from Postman .
And the collection:
Thus we will save a json from the environment and another from the collection in a folder, to which we will have to go from a command console to be able to execute Newman .
Now, we can choose to run the entire collection:
Or just a folder (adding the -f argument):
The output returns almost the same as Postman's: The execution of each request and a summary indicating how many tests failed and how many did not.
Jenkins
The last step to complete the test automation on the chosen API is to install Newman on the same server as Jenkins and install some plugins:
-
Log Parser Plugin : To parse the output.
-
Setting:
-
Description: Parsing error.
-
Parsing Rules File: File with the rules and the following content:
-
-
-
Email Extension Plugin : To receive an email if any test fails.
Once we have the plugins installed and configured, we create a new job (New job) with the following information:
Project name: Star Wars (for example)
Build → Add build step → Execute Shell and write the same series of commands that we wrote to execute Newman locally:
Post-build Actions → Add post-build action → Console Output (build log) parsing
And in Select Parsing Rules we select "Error Parsing" (Description)
Post-build Actions → Add post-build action → Editable Email Notification
And we fill in the following information:
-
Project Recipient List: Email addresses that should receive the email.
-
Advanced Settings → Add Trigger → Script - After Build
-
Trigger Script:
-
Save (Save) and Build (Build now) ...
The output console will show the same as when we run Newman locally along with some lines indicating that Jenkins has done its checks.
I have purposely run the "Starships" folder because I know that a test is going to fail and thus cause me to send an email. The email received will be:
In it we can see that the subject indicates that something went wrong (Something went wrong ...) and in the body of the message that Jenkins executed the job correctly and a link to the output of the execution to see the results.
Once we are in the job's output console, we can access “Parsed Console Output” to see the lines in which the tests that failed appear underlined in red.
Getting test automation on any API is as easy as following the steps indicated in this post, we only need to configure three programs and we will not have to worry about anything else. Jenkins will take care of sending us an email if a test failed.