Catch AI answers that got worse, slower or pricier.
On every pull request, through your real API.
Stresseur calls your AI feature the way users reach it: logged in, in the middle of a journey. It checks each answer's quality, speed and token use, and compares them with main.
AI checks need stresseur 0.4 or later.
$ npx stresseur ci
✓ ChatStreamsAnswer: 2 of 2 steps passed (12 runs, median 2.4 s, p95 2.9 s)
✓ StreamChat ttft 12 of 12, p95 610 ms ≤ 800 ms
✓ StreamChat latency 12 of 12, p95 2.8 s ≤ 6 s
✓ StreamChat tokens 12 of 12, max 288 ≤ 400
✗ StreamChat judge 8 of 12, score 3.4 (min 4): Answer omits where to cancel.
Judge: 8 calls, 4 cached, 1.2 s
AI features get worse without failing.
Status 200, valid JSON, and a worse answer. Three changes your tests miss today.
StreamChat › judge
97%→67%
8 of 12 runs passed
The answers got worse.
A prompt edit or a model swap. The endpoint still returns 200, but the answer skips a step or makes up a policy. A judge grades every answer, and Stresseur compares the pass rate with main.
StreamChat › ttft
610 ms→1.4 s
p95, budget 800 ms
The AI loop got slower.
The first token arrives later, or an agent makes one more tool call per answer. Stresseur times every step against main and checks your time-to-first-token budget.
AskAssistant › tokens
142→311
output tokens, budget 300
Each answer costs more.
A new model or a longer prompt can double the tokens per answer while the price sheet stays the same. Token budgets on the step catch it before the bill does.
One block on the step you already test.
Add expect: to the request that calls your AI. Your coding agent can write it: the Stresseur skill teaches it how.
- request: name: StreamChat depends_on: Login method: POST url: '{{ #env:APP_URL }}/api/chat' headers: { Authorization: 'Bearer {{ Login.response.body.token }}' }1 body: { message: 'How do I cancel my plan?' } stream: openai2 expect: max_ttft_ms: 8003 max_latency_ms: 6000 max_output_tokens: 400 judge:4 criteria: Answers how to cancel, using only the help doc. steps: - Check it says where to cancel (Settings > Billing). - Penalize any claim the help doc does not make. input: | Question: How do I cancel my plan? Help doc: Cancel any time in Settings > Billing > Cancel plan. output: '{{ response.text }}' min_score: 4- 1
Inside the journey
The token from the login step feeds the AI call. Recorded flows map this for you.
- 2
Reads the stream
Server-sent events are read as they arrive, so you get the full answer and the time to first token.
- 3
Speed and cost budgets
Time to first token, total time and output tokens, checked on every run.
- 4
A judge with fixed steps
Scores the answer from 1 to 5 against your criteria. The same steps every run, so it grades the same way.
Also: a JSON schema for structured answers, input and total token budgets, and streams from the common chat APIs and AI SDKs. The flow itself comes from recording your app or importing a HAR file.
Quality, in the pull request.
CI measures each flow 12 times. Stresseur turns every check into a pass rate and compares it with the last 30 runs on main. One bad answer in 12 is noise, not a drop.
PR comment
Updated on each push
StreamChat › judge 97% → 67% · warn
| Flow | Change | Base → head | Slower steps |
|---|---|---|---|
ChatStreamsAnswer | +2% | 2.35 → 2.40 s |
Gate: warns > +10%, fails > +30% (default) · a91c3e2 · median of 12
Quality
| Check | Pass rate | Base → head |
|---|---|---|
StreamChat › judge | −30 pts | 97% → 67% (8/12) |
StreamChat, iteration 3: “Answer omits where to cancel.”
Quality gate: warns ≥ 10 pts, never fails · base: main, last 30 runs
Warns first. Fails when you say so.
A drop of 10 points or more warns on the check. Set a fail_drop in your gate file to fail the check instead. Failing needs the Team plan.
Ask why it dropped.
Comment @stresseur why did quality drop. It answers with the pass rates, the failing runs and the judge's own reasons, each linked to the run it came from. Included on Team.
Why not just evals?
Eval tools like promptfoo, Braintrust and LangSmith test the prompt, and they do it well. Stresseur tests the endpoint you shipped, inside the user's journey, with speed and cost measured on the same runs.
- What it calls
- Evals
A prompt, or a function you wrap - Stresseur
Your real endpoint, logged in, after the steps before it - Where cases come from
- Evals
Written by hand, per prompt - Stresseur
Recorded from real use, or a HAR file - Speed and cost
- Evals
Limits per call - Stresseur
Limits per call, plus each step compared with main - Best at
- Evals
Many inputs and fine scores while you design a prompt - Stresseur
Catching a drop in the shipped feature before merge
Use both. Evals while you shape the prompt, Stresseur on every pull request after that.
Your key. Your CI.
Cents per pull request.
The judge runs in your CI with your own model key. Pick a small model and a pull request with 120 judged answers costs under 10 cents.
- ~$0.0007
- per judged answer, on a small model.
- $0
- for an answer that didn't change. Verdicts are cached, so it is never judged twice.
- 1–1.3 s
- per judge call, after the timed runs. Judging never counts in your latency.
- iterations:
- sets how many times CI calls your own AI per flow, so you control that bill too.
Know before you merge.
Test the AI feature your users actually reach, on every pull request. Free plans warn when quality, speed or cost drops. Team can fail the check. Start free for one repo.