LLM Evaluation in CI: Test Every Pull Request

LLM evaluation in CI: cheap checks on every pull request, a judge only where needed. See what to check, what a judge cannot tell you, and how to wire it.

Mustafa BayramogluMustafa BayramogluA glass prism splitting a beam of light across a row of small brass gauges

LLM evaluation measures whether your model, or the application built on it, behaves as intended. To do that in CI, put cheap checks on every pull request (the answer’s shape, its speed, its token use), add a judge model only for the questions those checks cannot answer, and compare pass rates with the main branch instead of trusting a single run. The rest of this guide covers what each layer can and cannot tell you.

What LLM evaluation means for a feature you ship

Arize describes LLM evaluation as the practice of measuring whether a large language model, or an application built on one, behaves as intended. For a feature you ship, the second half of that sentence is the part that matters. Your users never meet the raw model. They meet your prompt, your retrieval, your guardrails and the endpoint that wraps them.

That endpoint is an API, and an ordinary API test has a problem with it. A normal test sends a request and asserts on the answer. An AI answer changes from run to run, so assertions: can only check its shape. An endpoint can return a 200 and a well-formed body while the answer skips a step or makes up a policy. The status code stays green and the feature is broken. For a chat window, Chatbot Testing Through Your API applies the same idea to one endpoint.

So evaluation has to move one level up. Instead of one assertion per answer, you want a set of checks that each ask a narrow question, and you want to run them often enough to see a trend. A pull request is the natural place, and the same flow that runs your API smoke tests can carry the AI call. A prompt edit, a model swap and a retrieval change all arrive as a diff, and each can change the answers without touching a line of ordinary application code.

The sections below go in order of cost. Checks that need no model come first, because they are cheap and rarely flaky. A judge model comes second. Pass rates tie them together.

Model evals and product evals are different jobs

Search for “LLM evaluation” and two different activities share the results. The first is model evaluation. It assesses the overall quality of foundation models, such as GPT-4 or Llama 2, across a variety of tasks. Codesmith uses the same term for that model-level work. Google’s AI Overview for the same query puts it as testing the raw capabilities of a base model.

The second is product evaluation: testing how well an entire application, including prompts, retrieval, tool calls and guardrails, performs a specific real-world task. This is the one your pull request needs.

The distinction decides what you run in CI. A model benchmark tells you which model to start from. It cannot tell you whether your system prompt still produces the refund policy your support team wrote last week. Only a check against your own endpoint, with your own question and your own reference facts, can.

It also decides what you can fix. If a product eval fails, the cause is in something you changed: a prompt, a retrieval step, a model version you pinned or stopped pinning. That makes a failing check actionable in a way a benchmark score is not. When the thing that changed is a prompt, Prompt Testing on Every Pull Request shows how to run these checks on each prompt edit.

DeepEval, for one, describes itself as the open-source LLM evaluation framework for testing and benchmarking LLM applications. If you want a dataset-driven evaluation workflow, a framework like that is a reasonable choice. The approach in this guide is narrower: treat the AI call as one request in the flow your CI already runs, and check it the same way you check the rest of the API.

Checks that need no judge: shape, speed and tokens

Start with the checks that involve no second model. Google’s AI Overview lists them as code-based checks and heuristics: fast, programmatic tests that verify output format, length, JSON validity or keyword presence. They are cheap, deterministic and fast, so they can run on every push.

With Stresseur you add them as an expect: block on the request step that calls your AI. The block supports a JSON schema (schema, with schema_at to validate a path instead of the whole body), max_latency_ms, max_ttft_ms for streams, and token budgets (max_input_tokens, max_output_tokens, max_total_tokens, read from usage). The checks run after the flow’s steps, so they never count in its timings.

Each one catches a different failure:

  • Schema. If your feature asks the model for JSON that the rest of your code parses, a schema check fails the pull request the moment a prompt edit changes the shape. This is the most common silent break in structured output.
  • Latency. A longer prompt or a larger model can make the whole answer slower. A limit on the step’s duration turns that into a failed check before it becomes a user complaint.
  • Time to first token. For a chat feature the user waits for the first word, not the last. max_ttft_ms checks that wait, and it needs a stream: setting on the step.
  • Token budgets. A prompt that grows or a model that gets chattier raises cost on every request. A budget on input, output or total tokens catches that in review.

Streaming needs one extra setting. With stream: openai | anthropic | vercel-ai | sse the runner reads server-sent events as they arrive and exposes response.text, response.ttft_ms, response.usage and response.events. A later check can then read the assembled answer or the time to first token from the step like any other response field.

Here is the shape of a step with limits and no judge, taken from the example in the docs:

- request:
    name: StreamChat
    method: POST
    url: '{{ #env:APP_URL }}/api/chat'
    stream: openai
    expect:
      max_ttft_ms: 800
      max_latency_ms: 6000
      max_output_tokens: 400

The numbers in that block are the docs’ example values, not recommendations. Pick yours from what your feature does today, then tighten them as you learn the normal range.

These checks answer “did the answer arrive in the right shape, in time, at the expected cost”. They do not answer “was the answer good”. For that you need a judge.

When to use an LLM as a judge, and what it cannot tell you

Using one language model to grade another’s outputs has become increasingly popular, as the Hugging Face LLM course notes. The reason is that meaning cannot be matched with a regular expression. “Cancel any time in Settings” and “You can cancel from the billing page” say the same thing in different words, and only another model reads them as equal.

In Stresseur the judge is a G-Eval check inside expect:. It takes criteria, steps, input, output and a min_score from 1 to 5. The docs example asks whether the answer says how to cancel, using only the help document.

Two habits make a judge more useful and less noisy:

  1. Put the facts in input. The judge sees only input and output, so the question and any reference facts must be in input. A judge with no reference cannot tell a correct answer from a confident wrong one.
  2. Always write steps. They give the judge a fixed procedure to follow on every run, instead of leaving it to improvise.

The judge runs in your CI with your own key (STRESSEUR_JUDGE_API_KEY). Stresseur measured about 1.0 to 1.3 seconds per judge call and about $0.0007 per check on a small model, and unchanged answers are cached in .stresseur/judge-cache.json, so they cost nothing. To keep the number of calls, and so the cost, down, set iterations: 4 on the flow, which calls your AI fewer times per CI run. Those are measurements from Stresseur’s own runs, not a promise for your model and your prompt. You choose the judge’s model with judge: { provider, model } in the flow file.

Now the limits, which matter as much as the setup. A judge is itself a model. It can be wrong, and it can be wrong in a consistent direction. Treat its score as a signal that a person samples, not as a measurement of accuracy. A judge that passes everything is no more useful than a schema check that accepts any object, so read some of the answers it passes and some it fails, and rewrite the criteria when they disagree with you.

A judge also grades what you tell it to grade. If your criteria ask whether the answer mentions Settings, the judge will pass an answer that mentions Settings and gets everything else wrong. Write criteria that name the failure you fear, and add a second check for the next one.

Gate on pass rates, not on one run

Because an AI answer changes between runs, one failed run proves little. A pass rate across several runs says more. stresseur ci turns each check into a pass rate across the measured runs.

On the pull request, Stresseur Cloud compares that pass rate with the last 30 default-branch runs and adds a Quality section to the PR comment. The comparison is against your own main branch, so a feature that is a little flaky on main does not fail every pull request for being a little flaky. Only a change relative to the baseline shows up.

The gate warns at a 10-point drop. It fails only when you set gate.quality.pass_rate.fail_drop, which needs the Team plan or above. A sensible start is to leave it on warning, read the Quality section on a few real pull requests, and decide later whether a drop of that size should block a merge.

The same idea applies if you use another tool. Whatever runs your checks, ask three questions of the gate. Does it compare against main instead of a fixed number? Does it warn before it blocks? Does it show which check dropped, so the author knows where to look? A gate that only says “quality failed” teaches people to ignore it. The same applies to flaky tests: a signal that cannot name its cause loses trust fast.

Wire it into CI with a flow

Install the CLI and the recorder in the project:

npm install -D stresseur stresseur-recorder

AI checks need stresseur 0.4 or later. The first step is a flow that calls your AI. Flows come from recorded traffic or from a HAR file (generate --har <file>), with variables mapped so the login token feeds the AI call.

Then add expect: to the step. This is the docs’ full example, with a streamed chat answer, limits and a judge:

- request:
    name: StreamChat
    depends_on: Login
    method: POST
    url: '{{ #env:APP_URL }}/api/chat'
    headers: { Authorization: 'Bearer {{ Login.response.body.token }}' }
    body: { message: 'How do I cancel my plan?' }
    stream: openai
    expect:
      max_ttft_ms: 800
      max_latency_ms: 6000
      max_output_tokens: 400
      judge:
        criteria: Answers how to cancel, using only the help doc.
        steps:
          - Check it says where to cancel (Settings > Billing).
          - Penalize any claim the help doc does not make.
        input: |
          Question: How do I cancel my plan?
          Help doc: Cancel any time in Settings > Billing > Cancel plan.
        output: '{{ response.text }}'
        min_score: 4

Add the judge key as a CI secret with gh secret set STRESSEUR_JUDGE_API_KEY. Without a key, judge checks are skipped with a notice that names that command. The same happens on pull requests from forks, which get no secrets. A skipped check never fails the run, so a pull request from a fork can pass without the judge having run. Review those by hand.

The guide to API tests on every pull request covers the workflow itself. The AI testing page shows the Quality section, and the AI checks docs list every key.

What CI evals do not replace

A pull request check is a regression net. It tells you that a change moved a check you wrote. It does not tell you your feature is good, and it does not find the question you did not think to ask.

Three things stay outside it. Offline benchmarks and dataset-driven evaluation stay with your evaluation framework, if you use one. Reading real conversations stays with a person. And the numbers: Stresseur makes no accuracy claim for any judge, and its judge runs in your CI with your own key. What it gives you is the checks, the pass rates and the comparison with main, on the same pull request as the code that changed them.

Where to start

Add one flow that calls your AI, and put the cheap expect: limits on it first. Add a judge for the one question a schema cannot answer, with fixed steps and the reference facts in input. Leave the gate on warning until you have seen its Quality section on a few real pull requests.

Frequently asked questions

What is the difference between LLM evaluation and LLM testing?

The two terms overlap. LLM evaluation is the practice of measuring whether a model, or an application built on one, behaves as intended. LLM testing covers the accuracy, robustness, safety and reliability of language models before and after deployment. In CI the useful split is by cost: checks that need no model, and checks that need a judge.

Can I run LLM evals without a judge model?

Yes. A schema check, a latency limit, a time-to-first-token limit and token budgets all run without a judge. They check shape, speed and cost, not whether the answer is good.

Does Stresseur host the judge?

No. The judge runs in your CI with your own key, set with `gh secret set STRESSEUR_JUDGE_API_KEY`.

What happens to judge checks on pull requests from forks?

They are skipped with a reason and never failed. Pull requests from forks get no secrets, so there is no key for the judge to use.

Which model does the judge use?

You choose it at the top of the flow file with `judge: { provider, model }`. The provider is `anthropic` by default or `openai`, and any OpenAI-compatible API works with `base_url`.

All posts