Turn an OpenAPI Spec into Contract Tests

Turning an OpenAPI spec into contract tests takes two separate steps: generate schema-driven tests against the spec with a tool like Schemathesis, then add consumer-driven contract tests with a tool like Pact for any service that actually calls another one. The first step checks a service against its own documented spec. The second checks whether two services still agree on what that spec means once they talk to each other for real, which is the part an OpenAPI spec alone cannot answer.

The procedure: from OpenAPI spec to runnable tests

Three steps turn an OpenAPI spec into a working test suite:

1. Validate the spec itself. A malformed or incomplete OpenAPI document produces malformed tests, so lint it first with any OpenAPI validator before generating anything from it.
2. Generate schema-driven tests against one service. Point a property-based tool at the spec and let it build requests from the declared parameters, types and constraints. Schemathesis is the tool most often named for this step: it reads the OpenAPI spec directly and generates positive and negative requests to check for crashes, validation bypasses and schema violations.
3. Add consumer-driven contract tests where services actually call each other. A schema-driven test proves one service matches its own spec. It says nothing about whether the services that call it agree on what that spec means in practice, which is a separate problem contract testing solves.

The first step is mechanical. The second is where most teams start and stop, because a single OpenAPI spec is enough input to get real coverage for free. Schemathesis in particular needs almost zero manual effort beyond the spec itself. Schemathesis is one entry in a wider field; see Best AI Tools for API Test Generation for how it compares with tools built for other starting points. The third step is the one this piece spends the most time on, because it is the one an OpenAPI spec alone cannot give you.

For a single service with no downstream consumers, step 2 is often enough on its own. For a microservices setup where one service’s response shape is another service’s input, step 3 is not optional: two services can each pass their own schema-driven tests and still break each other the moment one changes a field the other depends on.

OpenAPI vs Swagger, and why it matters here

OpenAPI is the specification format itself: the language-agnostic, official way to describe a REST API’s endpoints, parameters and responses. Swagger is a specific set of open-source and commercial tools, built by SmartBear, for working with that specification.

In practice the two names get used interchangeably, and most files still called a “Swagger spec” or swagger.json in an older codebase are OpenAPI documents under a different name; the format was called Swagger before it was donated and renamed OpenAPI. Every tool in this piece, Schemathesis included, reads OpenAPI directly. If a spec file was generated by SwaggerHub or Swagger Editor, it is still an OpenAPI document and needs no conversion before the procedure above works on it.

Is Swagger outdated? The name is, for the specification: it was called the Swagger Specification up to version 2.0, and SmartBear donated it to the Linux Foundation in 2015, where it was renamed OpenAPI. The tooling is not outdated by that change. Swagger UI, Swagger Editor and Swagger Codegen are still products built to design, build and document APIs using the OpenAPI standard.

Is OpenAPI YAML or JSON? Either. An OpenAPI document is written in JSON or YAML to define endpoints, parameters and responses. Pick whichever your team finds easier to review in a pull request.

Schema-driven testing vs contract testing: where one ends and the other begins

Schema-driven testing checks one service against its own spec. Schemathesis is the clearest example: given an OpenAPI document, it generates thousands of positive and negative requests automatically and checks the service for crashes, data validation bypasses and schema violations. Nothing about this step requires a second service to exist. It answers one question: does this service actually behave the way its own spec says it does?

Consumer-driven contract testing checks whether two services agree on what the spec means. Pact is the tool most often named for this: it uses the Consumer-Driven Contract (CDC) approach, letting services be developed and deployed independently without a live end-to-end integration environment to test them together. This answers a different question: if service A calls service B, does B’s actual response still match what A is built to expect, even after B’s own team changed something?

Those two questions can both have a “yes” answer at the same time as a real break. Service B can pass every schema-driven test Schemathesis throws at it, because Schemathesis only checks B against B’s own spec. If B’s team then changes its response shape in a way that A never account for, A can break in production the next time it calls B, and neither team finds out until then. A schema-driven test never crosses the service boundary to check that.

This is why a microservices setup generally needs both, not one instead of the other. Schema-driven testing catches a service violating its own contract with the world. Contract testing catches two services silently disagreeing about what that contract actually says. The first step in the procedure above (schema-driven generation) is a prerequisite for a healthy service, not a substitute for the second (consumer-driven contracts) once that service has other services depending on it.

A worked example: one spec, start to finish

Take a small orders service with one relevant endpoint: POST /orders, which accepts a customerId string and an items array, and returns a 201 with an orderId and a status field. Its OpenAPI spec declares those types, marks customerId and items as required, and documents a 400 response for a missing field.

Step 2, schema-driven generation. Pointed at that spec, a tool like Schemathesis automatically reads the declared parameters and types from the spec and generates requests the developer would not necessarily think to write: an empty items array, a customerId of the wrong type, an oversized payload, a request missing items entirely. If the service returns a 500 instead of the documented 400 for that last case, or accepts a customerId it should reject, the generated test catches it without anyone having written “test malformed customerId” by hand.

Step 3, consumer-driven contract. Now say a billing service calls POST /orders after a purchase completes, and depends on getting back an orderId it can attach to an invoice. A contract test from the billing side (the consumer) records that expectation: given a valid request, billing expects a 201 with a non-empty orderId string. The orders team (the provider) then verifies their real service still satisfies that exact expectation, independently, without billing needing to run against a live orders instance to check it.

What the schema-driven pass in step 2 cannot catch: if the orders team renames orderId to id while keeping the spec’s field name unchanged in a rushed edit, or if the actual response drifts from the documented one in a way the spec was never updated to reflect, a schema-driven test checks the response against the written spec, not against what billing is actually depending on. The contract test in step 3 is written against the real expectation, not the spec document, so it catches exactly this kind of drift between what is documented and what a specific consumer actually needs.

What the spec can’t tell you

Everything in the procedure above starts from the same document: the OpenAPI spec. That is also its limit. A spec describes shape: field names, types, which fields are required, which status codes are documented. It does not describe what a service actually experiences once real clients start calling it.

A spec does not say how a client behaves when its auth token expires mid-request, or what payload shape a mobile client sends that a web client never would, or which endpoint sequence a real checkout flow actually calls in practice versus what the spec’s examples show. Schema-driven tests generate plausible-looking requests from the spec’s own rules, and contract tests check an agreed expectation between two services, but neither one is built from what real traffic actually looks like, because neither has access to it. The spec was written by a person describing intent, not recorded from a client doing the thing.

Stresseur’s approach starts from the other end of that gap: instead of generating tests from a hand-written spec alone, it learns how an API is actually used from real traffic. That is a different input, not a replacement for either technique above. A schema-driven test still catches a service violating its own documented contract. A consumer-driven contract test still catches two services disagreeing about what a field means. What neither one sees is the auth token that expired mid-flow in production last Tuesday, or the payload shape a specific client actually sends, because neither is looking at production traffic. That gap between what a spec can describe and what a service actually does under real use is the reason spec-based testing and usage-based testing answer different questions, not the same one answered two different ways.

An OpenAPI spec gets a service most of the way to real test coverage on its own: schema-driven generation with a tool like Schemathesis needs no hand-written test cases, only the spec. Where a spec runs out is the boundary between services, which is what consumer-driven contract testing with a tool like Pact is built to check. Between the two, a microservices setup gets coverage for one service’s own behavior and for what its neighbors actually depend on. What neither one sees is how the service behaves under real traffic once it ships, which is a separate problem with a separate answer.