The Benefits of Consumer Contract Driven Testing for REST APIs
7 minutes read (1698 words)
October 16th, 2023
Molasses Corp has over 100 different services that communicate via REST APIs. As the company grew, managing end-to-end integration tests became a nightmare. With multiple languages and frameworks, tests took hours to run and were flaky. The testing burden slowed down development for the entire organization.
When a backend team refactored a core service, they inadvertently broke downstream applications. Because end-to-end tests took so long to run, the issues weren't caught immediately. The outages were expensive both in terms of revenue and reputation for Molasses Corp.
To address these issues, Molasses adopted consumer contract driven testing (CCDT) which focuses on verifying integration contracts. By generating test SDKs from provider API definitions and running fast contract verification tests, issues are found within minutes versus hours or days. In fact, it changed the culture at Molasses Corp so much that they renamed their company to Warp Speed Ltd.
This story was fictionalized to protect the innocent but this is an all too familiar scenario that plays out across software enterprises. A familiar story such that the engineers at Referential Labs have helped multiple teams and organizations turn around their software delivery feedback cycles using techniques like CCDT.
In this article, we explain the benefits of CCDT that companies like Molasses Corp (now Warp Speed Ltd) realize when scaling complex microservices architectures. Here we will cover:
- background on the consumer-provider model in REST APIs
- the core concepts of CCDT
- the pros and cons of implementing CCDT
- best practices for adopting CCDT in your organization's REST API testing strategy
Continue reading to learn how to release APIs to external users with measured confidence that contracts are not violated.
API Contracts
A key to smooth integration between API consumers and providers is a defined contract. The contract specifies the interface between the services including endpoints, request/response formats, and more.
There are two types of contracts in CCDT:
- Consumer Contract - Defines the schema and endpoints that a consumer expects from a provider API.
- Provider Contract - Defines the schema and endpoints that a provider guarantees to implement for consumers.
For example, consider a provider API with the following OpenAPI contract:
openapi: 3.0.0
servers:
- url: https://api.server.com
paths:
/users/{id}:
get:
parameters:
- name: id
in: path
required: true
schema:
type: integer
responses:
'200':
description: User response
content:
application/json:
schema:
type: object
properties:
name:
type: string
email:
type: string
This contract defines the /users/{id} endpoint along with the request parameters and response format the provider guarantees to implement. The consumer contract would define what endpoints and schemas the consumer expects to utilize.
The provider OpenAPI contract above defines a GET /users/{id} endpoint for retrieving user data. Let's look at how this can be utilized in a consumer integration:
Consumer Contract
The consumer generates a client SDK from the OpenAPI contract to call the provider API. The SDK allows making requests and validating responses based on the contract.
// Generated SDK code
const api = new ProviderAPI({url: 'https://api.server.com'})
async function getUser(id) {
const response = await api.users.get({id});
// Validate response matches schema
response.validate();
return response.data;
}
The consumer relies on the SDK to automatically handle serialization, requests, and response validation based on the provider contract.
Provider Implementation
The provider implements the API endpoints and business logic:
// Provider code
app.get('/users/:id', (req, res) => {
const user = db.findUser(req.params.id);
return res.status(200).json(user);
});
Contract Verification
CCDT validates that the provider's implementation meets the contract. Example using a test validation server:
POST /validate
{
"request": {
"method": "GET",
"path": "/users/123"
},
"response": {
"status": 200,
"body": {
"name": "John Doe",
"email": "john@doe.com"
}
}
}
// Assertion - response matches schema
This validation ensures the provider meets the consumer's expectations.
Contract Mismatches
A contract mismatch occurs when the actual payload sent between a consumer and provider no longer matches the defined contract. Mismatches lead to integrations breaking when contracts fall out of sync.
Let's say, Company X recently launched a new partner integration consuming a vendor's API. A month after launch, the vendor added a new required "id_code" field to API requests. But Company X's integration was not updated to handle this change.
This type of mismatch can result in broken integration flows rejecting requests.
There are two common types of contract mismatches:
- Intended
- Knowingly changing the contract and payload (e.g. adding a field)
- Unintended
- Unknowingly changing the payload structure (e.g. due to refactoring)
This hypothetical demonstrates how even intended contract changes can lead to unintended consequences if consumers are unaware. CCDT provides greater confidence by verifying contracts early.
Testing Benefits
End-to-end (E2E) testing is commonly used today to validate REST API integrations. However, as the number of services grows, manually orchestrating and executing E2E tests becomes increasingly complex, slow, and expensive. E2E tests are difficult to maintain across teams, lead to slow feedback cycles, and provide poor coverage of edge cases. By moving contract verification earlier in the development lifecycle with CCDT, it can provide faster feedback on integration issues through lightweight unit tests, reducing the necessary footprint of long feedback cycle E2E tests for the same API coverage. Shifting left on contract testing is important for organizations looking to scale their REST APIs across internal and external consumers. CCDT complements ongoing E2E testing efforts by verifying contracts early, enabling E2E tests to focus on business logic and user flows over contract mismatches.
Understanding CCDT
Let's continue the example from the contract mismatches section to understand CCDT.
Company X defined a consumer contract specifying the API schema and endpoints they required from the vendor:
"/users": {
"get": {
"request": {
"query": [
"limit",
"offset"
]
},
"response": {
"body": [
"id",
"name"
]
}
}
}
However, the vendor had a different provider contract:
"/users": {
"get": {
"request": {
"query": [
"limit",
"offset",
"id_code" // Added later
]
},
"response": {
"body": [
"id",
"name"
]
}
}
}
The mismatch in required query parameters led to integration failures.
With CCDT, the vendor could have validated their provider contract changes against Company X's consumer contract. This would have caught the breaking change before impacting consumers.
CCDT gives providers confidence that they are satisfying all consumer expectations as contracts evolve on both sides. This is critical for preventing unintentional breaks.
Implementing CCDT
There are better practices for effectively implementing CCDT:
Generate consumer SDKs from provider contract
Publish OpenAPI (formerly Swagger) specifications that define your public API contract.
Consumers use tools like Swagger Codegen to automatically build SDKs in languages from those definitions. The SDKs handle serialization and response validation - a boost for consumers.
Use provider integration tests for contract verification
Run provider integration tests against the consumer contract to verify the provider contract. This ensures the provider meets consumer expectations.
At one ecommerce SaaS customer of ours, we leveraged CCDT to verify their provider contract for their integration partners. They had a suite of integration tests that ran against their API. We refactored the test suite to run most tests against the consumer contract in a lightweight and lightening fast test environment verifying the provider contract retaining test coverage while drastically reducing the time-to-feedback from their CI pipeline from multiple hours to ten minutes.
Test common response codes
When testing an API, focus on the shape of responses rather than internal logic. We prefer to put together representative requests for common response codes like 200, 400, and 500. Different response codes can use different schemas, so testing format per code is important.
Test common request parameters and payloads
Test common request parameters like query parameters, headers, path parameters and request and response payloads with the minimum amount of structure possible. These are the most common sources of contract mismatches.
CCDT in Action
Let's walk through how Company X from our previous examples could have avoided their vendor API integration issues by leveraging CCDT.
Defining the Contracts
First, Company X needs to define clear consumer-driven contracts specifying their expectations for the vendor's API. These contracts document the required endpoints, parameters, schemas, etc.
For example, they may define the following OpenAPI contract for the /users endpoint needed:
# Company X's consumer contract
"/users":
get:
parameters:
- name: limit
in: query
- name: offset
in: query
responses:
200:
description: success response
content:
application/json:
schema:
type: array
items:
type: object
properties:
id:
type: integer
name:
type: string
Meanwhile, the vendor defines their provider contract implementing the actual API:
# Vendor's provider contract
"/users":
get:
parameters:
- name: limit
- name: offset
- name: id_code
responses:
200:
description: success response
content:
application/json:
schema:
type: array
items:
type: object
properties:
id: integer
name: string
Verify Contract Compliance
As API changes are made, Company X validates their API responses against consumer contracts using CCDT.
For example, when the id_code query parameter gets added as required by the API team, CCDT should catch this contract breaking change. The API can then decide to handle the new parameter via a default behavior and make the query parameter optional avoiding breaking the consumer contract.
CCDT gives Company X higher confidence that their API won't break consumers unexpectedly due to undocumented API changes whether intentional or not.
CCDT in CI/CD Pipeline
Company X hooks up their CCDT validation as part of automated testing in their CI/CD pipeline. Now contract compliance is checked on every code change before reaching production.
CCDT shifts API testing left and enables faster feedback due to its lower overhead. This is critical for enterprise microservice architectures.
Wrapping up
Consumer contract driven testing provides a useful technique for verifying integration contracts between API providers and consumers. The benefits for organizations like Company X are:
- Improved efficiency through lightweight contract verification
- Better scalability by avoiding brittle and slow end-to-end setups
- Faster feedback on changes impacting consumers
- Confidence in releasing API changes
As APIs become central to modern software, CCDT is important for maintaining quality and reliability across services. API teams could improve productivity by adopting CCDT practices like:
- Defining clear consumer-driven and provider-driven contracts
- Implementing contract validation in your CI/CD pipelines
- Shifting testing left to gain rapid feedback
If you want help streamlining any part of your software delivery and shift feedback left in any way, contact us at Referential Labs to find out if and/or how we could help your organization.