Chatbot Testing Through Your API: What to Check
Chatbot testing through your API: check answer shape, speed, tokens and quality on every pull request, with a judge only where simple checks fall short.

Chatbot testing through your API means testing the endpoint the chat window calls, not the window itself. Send it a question from a CI flow, then check the answer’s shape, its speed, its token use and, where those are not enough, its quality with a judge model. The first three need no second model. The rest of this guide shows which check catches which failure and where the checks stop.
Test the endpoint behind the chat window
A chatbot looks like a chat window, but the window is a thin layer. Behind it sits an API call that carries the user’s message and returns an answer. That call is the part you can test on every pull request, from the same CI job that already tests the rest of your API.
The catch is that the answer changes from run to run, so assertions: can only check its shape. An endpoint can return a 200 and a well-formed body while the answer is wrong, so the status code alone proves little. When the bot’s behavior comes from a prompt, Prompt Testing on Every Pull Request shows how to run the same checks on every prompt edit.
In Stresseur you add an expect: block to the request step that calls your AI. The checks run after the flow’s steps, so they never count in the flow’s timings. The sections below go in order of cost: checks that need no model, checks for streamed answers, then a judge.
Checks that need no judge: shape, speed and tokens
Start with the checks that involve no second model. They are cheap and they fail for clear reasons.
- Schema.
schemachecks the body against a JSON Schema, inline or from a file path.schema_atvalidates a path instead of the whole body. If your bot returns structured data next to its text, a prompt edit that changes that shape fails the pull request at once. - Latency.
max_latency_mschecks the step’s duration. A longer prompt or a larger model shows up as a failed check instead of a user complaint. - Tokens.
max_input_tokens,max_output_tokensandmax_total_tokensare token budgets. A bot that turns chattier after a change costs more on every request, and a budget catches that in review.
A step with limits and no judge looks like this:
- request:
name: Chat
method: POST
url: '{{ #env:APP_URL }}/api/chat'
body: { message: 'How do I cancel my plan?' }
expect:
max_latency_ms: 6000
max_output_tokens: 400
The numbers are example values, not recommendations. Pick yours from what the bot does today, then tighten them.
Streaming chatbots: time to first token
Most chat features stream the answer, so the user waits for the first word, not the last. Put stream: on the request step with openai, anthropic, vercel-ai or sse for a raw stream. The step then exposes response.text, response.ttft_ms, response.usage and response.events. With the stream in place, max_ttft_ms checks the time to first token, and it needs stream:.
stream_timeout_ms defaults to 30000 and fails a stream that never closes. A recorded stream already gets the right stream:. That covers the failure where the connection opens, the bot hangs and nothing ever arrives.
- request:
name: StreamChat
method: POST
url: '{{ #env:APP_URL }}/api/chat'
body: { message: 'How do I cancel my plan?' }
stream: openai
expect:
max_ttft_ms: 800
max_latency_ms: 6000
max_output_tokens: 400
Again, the values are the docs’ example, not a target for your bot.
Grade the answer with a judge, and know its limits
Shape and speed do not say whether the answer is good. For that, judge is an LLM judge with criteria, steps, input, output and a min_score from 1 to 5. The judge sees only input and output, so put the question and any reference facts in input, and always write steps. A judge with no reference facts cannot tell a correct answer from a confident wrong one.
The judge runs in your CI with your key. Stresseur measured about 1.0 to 1.3 seconds per judge call and about $0.0007 per check on a small model, and unchanged answers are cached, so they cost nothing. Those are measurements, not a promise for your model and your prompt.
Having one language model assess another’s outputs has become increasingly popular, as the Hugging Face LLM course notes. A judge is itself a model. It can be wrong, and wrong in a consistent direction, so treat its score as a signal for a person to sample, not as a measure of accuracy. The guide to using an LLM as a judge covers where it fails. It also grades only what you tell it to grade: criteria that ask whether the answer mentions Settings will pass an answer that mentions Settings and gets everything else wrong.
Prompt injection and other cases to script
A chatbot takes free text by design, so odd input is the normal case. OWASP defines a prompt injection vulnerability as one that occurs when user prompts alter the LLM’s behavior or output in unintended ways. Do not hope it away. Script it.
Add a step that sends the hostile message, and a judge whose criteria name the failure you fear, such as the bot revealing its instructions. Put the rule the bot must keep in input, next to the question. Do the same for the typo, the slang and the off-topic question: one step per case, one criterion per failure.
Run it on every pull request
You need a flow that calls the chat endpoint. To start from a browser capture, use generate --har. npx stresseur ci then turns each check into a pass rate across the measured runs. AI checks are in the CLI from 0.4.0 and in Stresseur Cloud.
On the pull request, the comment gets a Quality section when a check’s pass rate drops against the last 30 runs on main. A drop of 10 points warns. One failed run of a non-deterministic bot proves little, and the comparison with main keeps a slightly flaky bot from failing every pull request.
Pull requests from forks get no secrets, so their judge checks are skipped. The LLM evaluation in CI guide covers the gate in more depth, the AI testing page shows the Quality section, and the AI checks docs list every key.
What this approach does not cover
Testing through the API checks what your endpoint returns. It does not check how the chat window looks or behaves in a browser, and a browser test of the widget is a different job. It does not replace reading real conversations either, because a check only asks the question you wrote down.
It also covers only the conversations you script. A flow sends the messages you put in it, so a bot that fails on a phrasing nobody thought of will pass until someone adds that phrasing as a step. Keep a short list of real failures from support tickets or logs, and turn each one into a flow step with a check once you have fixed it. Over time the flows become a record of what has gone wrong before, which is the most useful regression suite a chatbot can have.
Stresseur makes no accuracy claim and offers no hosted judge. So the pass rate tells you whether your checks moved, not how good the bot is. A falling pass rate is a reason to read the answers.
Where to start
Treat the chatbot as an endpoint. Put shape, latency and token limits on it first, add max_ttft_ms if it streams, and add a judge for the one question those cannot answer. Leave the gate on warning until you have seen the Quality section on a few real pull requests. None of this replaces reading real conversations.
Frequently asked questions
Do I need an LLM judge to test a chatbot?
No. The schema, latency, time-to-first-token and token checks are keys of their own, separate from `judge`, so you can use them alone. They check shape, speed and cost, not whether the answer is good.
Does Stresseur host the judge?
No. The judge runs in your CI with your key, which you add as a secret with `gh secret set STRESSEUR_JUDGE_API_KEY`.
What happens to judge checks on pull requests from forks?
Pull requests from forks get no secrets, so their judge checks are skipped and the notice says why. A skipped check never fails the run, so look at those pull requests by hand.
Can I test a chatbot that streams its answer?
Yes. Put `stream:` on the request step and it exposes `response.text`, `response.ttft_ms`, `response.usage` and `response.events`. A recorded stream, or one in a HAR file, already gets the right `stream:`.
How do I get a first flow for my chatbot?
To start from a browser capture, use `generate --har`. `generate` also prints a hint on AI endpoints, and you can add `expect:` yourself or let your coding agent do it after `install-skill`.