Prompt Testing on Every Pull Request
Prompt testing in CI: turn every prompt edit into a pull request check with fixed inputs, cheap limits and a judge. See what to run and what it misses.

Prompt testing means running the same fixed inputs against your prompt configuration every time it changes, and checking the answers against explicit expectations. On a pull request that is a flow that calls the endpoint running your prompt, with expect: checks for shape, speed and tokens, a judge for the questions those cannot answer, and a pass rate compared with main. A prompt edit is a diff like any other, so it gets a check like any other.
What prompt testing is, and why a pull request is the place for it
Braintrust defines prompt testing as running a prompt and its model configuration against representative inputs, attaching an explicit expectation to each case, and applying the same scoring method across every run. The word configuration matters. Prompt behavior depends on the prompt text, model, parameters, tools and retrieved context, so changing any of those can alter results even when the prompt itself stays similar. Every one of those changes arrives on a pull request as a diff, so that is where the suite runs.
The failures are quiet. MLflow’s examples are a prompt that returned clean JSON last week and might start wrapping its response in commentary, and a summarizer that might drift toward verbosity and drive up token costs. Neither throws an error. The endpoint still answers, and only a check on the answer notices.
Test the endpoint that runs the prompt
Your users never meet the prompt file. They meet the endpoint that wraps it, so the test goes on the request step that calls that endpoint, inside the same flow that runs the rest of your API. An AI answer changes from run to run, so assertions: can only check its shape. You add expect: to the request step that calls your AI. Its checks run after the flow’s steps, so they never count in the flow’s timings. For a streamed answer, stream: reads the events as they arrive and exposes response.text to later checks.
One consequence follows. The app in CI has to run the changed prompt. If the prompt lives in your repository, the pull request’s build already contains it. If it lives in a provider dashboard, a pull request cannot change it, and the check only sees whatever the endpoint serves that day. The AI testing page shows what the check looks like on a pull request, and Chatbot Testing Through Your API applies the same setup to one chat endpoint.
Build the case set: a small list of inputs worth keeping
A prompt suite is a list of inputs with an expectation attached to each. A strong suite covers normal cases, edge cases, known failures and adversarial inputs, so that an improvement on one example does not hide a regression on another.
In a flow, each input is one request step with its own expect: block. Name the steps for what they test, such as RefundQuestion or IgnoreYourInstructions. MLflow’s advice on labels fits here: keep perturbation pairs small and labeled by failure mode, so a failing test names what broke instead of only reporting that something did.
Start with a handful. Take the questions your feature exists to answer, one odd input (empty, very long, in another language) and one attempt to make the model ignore its instructions. Then grow the set from real failures. Braintrust describes adding production failures back to the test set, which keeps regression coverage aligned with what users hit. A bug report becomes a new request step in the pull request that fixes it.
Checks that need no judge: shape, speed and tokens
Begin with checks that need no second model. The expect: block takes a JSON schema (schema, or schema_at for one path), max_latency_ms, max_ttft_ms for streams, and token budgets (max_input_tokens, max_output_tokens, max_total_tokens, read from usage). They are cheap and they catch what prompt edits tend to move. A schema check is the direct answer to the first of those two failures, because commentary around the JSON breaks the shape. A token budget catches the summarizer that grows chatty. Braintrust adds that edits which add examples, context or reasoning instructions can improve accuracy while also increasing response time or spend, so those changes belong in the same test run.
- request:
name: RefundQuestion
method: POST
url: '{{ #env:APP_URL }}/api/chat'
body: { message: 'Can I get a refund after 30 days?' }
stream: openai
expect:
max_ttft_ms: 800
max_latency_ms: 6000
max_output_tokens: 400
The numbers are placeholders. Pick yours from how the feature behaves today, then tighten them. These checks say the answer arrived in the right shape, in time and at the expected cost. They do not say it was right.
Add a judge for the question a schema cannot answer
For “was it right”, add a judge. In Stresseur it is a G-Eval check inside expect: with criteria, steps, input, output and a min_score from 1 to 5. The judge sees only input and output, so put the question and any reference facts in input, and always write steps. For a prompt that must state a refund policy, the policy text goes in input. Without it the judge cannot tell a correct answer from a confident wrong one.
judge:
criteria: States the refund window from the policy and nothing else.
steps:
- Check the answer gives the window the policy states.
- Penalize any rule the policy does not contain.
input: |
Question: Can I get a refund after 30 days?
Policy: Refunds are available within 14 days of purchase.
output: '{{ response.text }}'
min_score: 4
The judge runs in your CI with your own key, set with gh secret set STRESSEUR_JUDGE_API_KEY. Stresseur measured about 1.0 to 1.3 seconds per judge call and about $0.0007 per check on a small model, and unchanged answers are cached and cost nothing. Those are measurements from its own runs, not a promise for your model and prompt. With no key, judge checks are skipped with a notice that names the command, and a skipped check never fails the run.
A judge is a model, and it can be wrong in a consistent direction. Treat its score as a signal that a person samples, not as a measure of accuracy: read some answers it passes and some it fails. The guide to LLM as a judge covers that in depth.
Read the result as a pass rate against main
One run of an answer that changes proves little. stresseur ci turns each check into a pass rate across its runs. In Stresseur Cloud that rate is compared with the last 30 default-branch runs, and the pull request comment gains a Quality section. The gate warns at a 10-point drop and fails only when gate.quality.pass_rate.fail_drop is set, on Team and above.
Comparing with main matters because a case that is a little flaky there should not fail every pull request. It also keeps the schedule short. MLflow suggests running the fast suite on every pull request and the fuller dataset comparison nightly in staging. Leave the gate on warning at first, read the Quality section on a few real pull requests, and only then decide whether a drop of that size should block a merge. The same logic applies to flaky tests: a signal that cannot name its cause loses trust.
What this does not replace
A pull request check is a regression net. It tells you that a change moved a check you wrote, and it cannot find the question you did not think to ask. Dataset-driven evaluation, reading real conversations and a judge you have calibrated yourself stay with you. For the full set of layers, see LLM Evaluation in CI, and the AI checks docs list every key.
Add one flow with five inputs, put the cheap limits on it first, and add a judge for the one question a schema cannot answer. The next prompt edit then arrives with its own result attached. The workflow itself is the one in API tests on every pull request.
Frequently asked questions
What is the difference between prompt testing and prompt evaluation?
Prompt tests supply the fixed cases and repeatable runs, while evaluation supplies the scoring framework used to interpret the results. On a pull request you need both: the cases come from your request steps, and the scoring comes from `expect:` checks and a judge.
Do I need a judge model to test prompts?
No. A schema, a latency limit, a time-to-first-token limit and token budgets all run without a judge. They check shape, speed and cost. Add a judge only for the questions they cannot answer.
Which model does the judge use?
You pick it at the top of the flow file with `judge: { provider, model }`. The provider is `anthropic` by default or `openai`, and any OpenAI-compatible API works with `base_url`.
What happens to judge checks on pull requests from forks?
They are skipped, and the notice says why, because pull requests from forks get no secrets. A skipped check never fails the run, so review fork pull requests by hand.
Can Stresseur test a prompt that has no endpoint in front of it?
No. The checks go on the request step that calls your AI, so the flow needs an endpoint it can call. A prompt that only runs inside a notebook or a provider dashboard is outside what a pull request check can reach.