Automatically generate contract tests from an Open API spec

[#1]

Before diving into OpenAPI contract testing, let's briefly discuss what is an OpenAPI specification. It is a specification for building APIs. It provides a standardized way to describe, document, and visualize RESTful APIs. The specification is written in YAML or JSON. It defines the endpoints, request parameters, and response structures. Using the OpenAPI Specification, developers can easily create API documentation that is both human-readable and machine-readable.

So what is OpenAPI Contract testing -

First, look at what is contract testing in general.  It's a technique for testing an integration point by checking each application in isolation. A specific testing that ensures a consumer can talk to the provider properly, by following a written "contract". To be more specific when a contract is written in the form of OpenAPI spec, it's referred to as OpenAPI contract testing.

Why OpenAPI contract testing -

Having contract tests in place enables asynchronous development of the client and server via the pre-established contract. Note while the server team focuses on retrieving the data, the client team can focus on presenting the data to the user and they both can do so in parallel. Since contract tests will enforce that their communication doesn't break during development, both teams don't have to wait on each other to validate the communication.

This is the fastest way to validate messaging between client & server, as you don't need to wait to build the system completely.

Another good side effect of having a contract in the form of OpenAPI specification is that it enforces that documentation stays current because documentation is part of the OpenAPI spec or contract.

An OpenAPI sample contract -

Let's say Autodesk’s Internal Development Portal has a REST endpoint that exposes software catalog data via a Catalog object, Catalog object is defined as below with a few fields.

Catalog
{
id: integer,
name: string,
description: string,
owner: string,
status: string
}  

The contract also defines return responses and error conditions and even headers required as shown below.

Method: GET 
Headers to use
Authorization: Bearer <OAuth Token>
Success Code: 200
Success Response: { < Catalog object> }
Error Responses
404 – { error: “Catalog not in database” }
401 – { error: “Unauthorized to make this request.” }

In a sample OpenApi specification, in YAML format, this will look something like the below image.

Also, note that one can specify what fields are mandatory. In the below example, id, name & owner are mandatory fields. 

 

Sample breach of contract -

Now once a contract has been established, whenever a break occurs, we want to know about it as part of the validation. For example below, note that due to some bug, now the server missed out "owner" field in the response.

Several available tools for automation -

There are several tools available to ensure automatic contract testing. Some of them are mentioned below

• Dredd
• Dredd is a language-agnostic command-line tool for validating API description documents against the backend implementation of the API. Dredd reads your API description and step by step validates whether your API implementation replies with responses as they are described in the documentation, including headers.
• Nswag
• This is a .NET library that can generate client code and unit tests for multiple programming languages from an OpenAPI specification. It supports C#, TypeScript, and other languages.
• OpenAPI-Testing
• This is another JavaScript library that can be used to generate unit tests from an OpenAPI specification. It uses the Mocha testing framework and allows you to write tests in JavaScript or TypeScript.
• Swagger-unit-test
• This is a Python library that allows you to generate unit tests from OpenAPI specification

Just a bit more about Dredd -

Dredd is a command-line tool that allows you to write tests in JavaScript, and it will automatically generate test cases based on your OpenAPI specification. It can also be used to test your API endpoints as you develop them. Dredd will look at the headers, status code, response format, and the exact response body returned, and let you know if there are differences in any of those areas. A detailed error message will be returned, meaning you can debug your code.

Concept of hooks -
Dredd has this concept of hooks, which can be used to do common things. It's like a mechanism for setup and tear-down for the tests. For example, if you want to do something common before all tests are run, say acquiring a bearer token. You can code that inside Hooks.

Sometimes you just need a specific test to have a specific setup. For example, if you have to test an API from OSS say GetBucketDetails(), and your test expects a bucket to be pre-created before it runs, then you can utilize the hook to first create a bucket.

Let's try Dredd hands-on -
Install Dredd -

$ npm install -g dredd

Run your server -

node app.js &

Test your running server - 

Note api-sample-server.yml is where you define the specification for your server. Or this is often called an OpenAPI spec file.

dredd api-sample-server.yml http://127.0.0.1:3000

This should return all test PASS in the console logs.

Now if you change the server code to not return the field "owner" and rerun your tests. You will see that Dredd will return a test failure with specific details about the "owner" field missing in the response.

Summary -
Right after bootstrapping the client & server projects, very quickly both teams can get busy adding features and end up breaking the contract between client & server. In order to keep both applications in sync automatically. Without investing a lot into hand-written code internally, teams can leverage open-source tools to automate the validation of contracts. 

Basically, this approach mandates you to start with API first, then document those APIs in OpenAPI specification format (which is both human and computer readable) after which you use contract testing tools to ensure contracts stay valid.  The documentation stays current throughout the life cycle of both products.

We at Autodesk are working on tools that can intake OpenAPI spec and automatically generate SDKs in various programming languages.

Key takeaways - 

• Validated contracts are crucial for correct & up to date documentation
• Saves time in the development process as client and server development can happen in parallel
• Contracts are both human-readable and machine-readable
• Allows for communication before API development starts
• Allows verification of applications in isolation