API Smoke Tests on Every Pull Request

An API smoke test is a short run of your most important API calls. Its purpose is to find simple but critical problems before in-depth testing begins. On every pull request, that means a small set of flows in CI that fails the check when the API cannot do its main job. This post shows how to pick those flows, run only that set in GitHub Actions, and make a failure block the merge. It builds on what API testing is and assumes your API can start in CI.

Set up an API smoke test in four steps

1. Pick the few flows that prove the API is alive, such as signing in and reading one resource.
2. Name them as a separate, short list, apart from the rest of your API tests.
3. Run only those flows in CI on each pull request.
4. Make the check required, so a red result stops the merge.

Stresseur’s ci command accepts flow files as arguments, so step 3 needs no custom script. The init-ci command accepts a flow directory and a workflow path, which gives a second way to get a separate smoke workflow.

Set your own time budget for the check. A smoke set earns its place by being fast, so if a flow makes the check slow, move it to the full suite.

What an API smoke test checks

Atlassian describes smoke tests as basic tests that check the basic functionality of an application and are meant to be quick to execute. Its aim is to find out whether the application is so badly broken that further immediate testing is unnecessary.

The questions are blunt. For software in general, Wikipedia lists questions such as “does the program run?” and “does clicking the main button do anything?”. For an API the same questions become: does the service answer, does a sign-in work, does the main read return data.

A smoke test checks the whole system together and is not meant to be exhaustive. That limit is deliberate. A passing smoke test says the build is worth testing further. It does not say the build is correct. The end-to-end testing tools guide shows where API flows like these sit next to browser tests.

Choose the flows for the smoke set

Choose flows by one question: if this fails, is there any point running the rest? Three kinds of flow usually pass that test:

• A sign-in that returns something the next request can use.
• One read of the resource your product is built around.
• One create, followed by a read of the item you just made.

Leave out error branches, permission matrices and odd inputs. They are useful and they belong in the full suite, ranked by what to automate first.

If you record your flows with Stresseur, the generated flows send fresh sign-up values each run and check each answer, as well as the status. That makes a short flow a stronger smoke check than a bare status-code ping, because a 200 with the wrong body still fails. If you already use Playwright for API checks, Playwright API Testing vs a Recorded Flow compares it with a recorded flow. The flows live in the repo under api-tests/ as *.yamlflow.yaml. You can try a single flow on your machine before it goes into CI:

npx stresseur flow run api-tests/checkout.yamlflow.yaml checkout

That runs the named flow. For how Stresseur records flows, see the API testing page.

Run only the smoke set on every pull request

Left alone, npx stresseur ci runs every flow in api-tests/ 13 times, one warm-up and 12 measured. To run only the smoke set, name the files:

npx stresseur ci api-tests/login.yamlflow.yaml api-tests/orders.yamlflow.yaml

The command takes the flow files as arguments, and also has --iterations and --warmup flags for the number of measured and warm-up runs.

The workflow that init-ci writes is .github/workflows/stresseur.yml, and on each pull request and push to the default branch it starts your app, runs the flows and uploads the timings. For a second, smoke-only workflow, either copy that file and name the smoke flows in the run step, or run npx stresseur init-ci --tests <dir> --out <path> against a folder that holds only the smoke flows.

The app starts from a root Docker Compose file, a Dockerfile or a package.json start or dev script, in that order. To test a deployed server, set the APP_URL repository variable to its URL. The CI setup guide covers the details. Each pull request then gets one check named Stresseur / performance and one comment that is updated in place on each push.

Run it first and let it gate the rest

Order matters more than people expect. A smoke test is useful right after a new build is made, to decide whether you can run more expensive tests. On a pull request, that makes the smoke job the first gate: if it is red, the slower suites have nothing worth reporting yet. If you have not picked a tool to run the smoke set, the guide to choosing an API testing framework compares the options.

Here is how that looks for a small shop API. The smoke set has three flows. The first signs in. The second lists products. The third places an order and reads it back. Everything else, such as refunds, permission checks and bad input, stays in the full suite. When the smoke job passes, the full suite runs. When it fails, the pull request shows one red check with one broken step, which is a short list to read.

Keep smoke flows from going flaky

A smoke check that fails for the wrong reason trains people to ignore it. Three habits keep its failures meaningful.

• Use fresh data. Recorded flows send fresh sign-up values each run, which removes one of the common causes of flaky tests.
• Keep secrets out of the flow files. Variables listed in .env.example come from repository secrets, and uploads use GitHub OIDC, so there is no token to store.
• Make it easy to reproduce. The pull request comment includes the command to run a flow locally, so the author can rerun the failing flow before pushing a fix.

Fail the pull request, and keep the set small

A failed step makes npx stresseur ci exit with code 1, and a non-zero exit fails the CI job. Mark that job as a required check in your branch protection settings, and a broken smoke flow stops the merge. The post on running API tests in GitHub Actions walks through the required-check setting.

Keep the two kinds of failure apart. The smoke job fails because a step broke. The speed gate is a separate rule, covered in catching breaking API changes in pull requests: without a file, it warns above +10% and fails above +30%, but on GitHub-hosted runners the default gate only warns. If a smoke flow is slow but correct, that is a speed question for the gate, not a reason to fail the smoke job.

If a smoke test fails, one explainer advises holding the deploy until the failure is fixed. Fix the smoke flow first, because the slower suites tell you little about a build that cannot do its main job.

Resist the urge to grow the set. Each flow you add makes the check slower and gives it one more reason to fail for a reason that is not about the build being alive. Depth belongs to integration tests, which check how the connected parts work together. For where the line sits, read smoke test vs integration test.

Conclusion

Pick the handful of flows that prove the API is alive, run only those on every pull request, and make the check required. Keep the set thin so it stays fast and its failures mean one thing: the build is broken. Everything deeper runs after it passes.