AI checks

AI checks need stresseur 0.4 or later.

An AI answer changes from run to run, so assertions: can only check its shape. Add expect: to the request step that calls your AI. The checks run after the flow’s steps, so they never count in its timings. npx stresseur ci turns each check into a pass rate across its runs.

Keys

Key	Checks
schema	The body against a JSON Schema, inline or a file path
schema_at	A path to validate instead of the whole body
max_latency_ms	The step’s duration
max_ttft_ms	Time to first token (needs stream:)
usage	Where the response reports tokens (default response.body.usage)
max_input_tokens, max_output_tokens, max_total_tokens	Token budgets
judge	An LLM judge: criteria, steps, input, output, min_score (1–5)

Example

- request:
    name: StreamChat
    depends_on: Login
    method: POST
    url: '{{ #env:APP_URL }}/api/chat'
    headers: { Authorization: 'Bearer {{ Login.response.body.token }}' }
    body: { message: 'How do I cancel my plan?' }
    stream: openai
    expect:
      max_ttft_ms: 800
      max_latency_ms: 6000
      max_output_tokens: 400
      judge:
        criteria: Answers how to cancel, using only the help doc.
        steps:
          - Check it says where to cancel (Settings > Billing).
          - Penalize any claim the help doc does not make.
        input: |
          Question: How do I cancel my plan?
          Help doc: Cancel any time in Settings > Billing > Cancel plan.
        output: '{{ response.text }}'
        min_score: 4

The judge sees only input and output, so put the question and any reference facts in input. Always write steps. Set iterations: 4 on the flow to call your AI fewer times per CI run.

The judge key in CI

The judge runs in your CI with your key. Add it as a secret:

gh secret set STRESSEUR_JUDGE_API_KEY

npx stresseur init-ci already maps STRESSEUR_JUDGE_API_KEY, ANTHROPIC_API_KEY and OPENAI_API_KEY from your secrets. It does so on the run step only, so your app never sees them. An empty secret counts as not set.

Without a key, judge checks are skipped with a notice that names gh secret set STRESSEUR_JUDGE_API_KEY. A skipped check never fails the run.

Pull requests from forks get no secrets. Their judge checks are skipped, and the notice says why.

Pick the model at the top of the flow file with judge: { provider, model }: provider is anthropic (the default) or openai, for any OpenAI-compatible API with base_url. Verdicts are cached in .stresseur/judge-cache.json, so an unchanged answer is never judged twice.

Streaming

Put stream: on the request step: openai, anthropic, vercel-ai or sse (raw). The step then exposes response.text (the assembled answer), response.ttft_ms, response.usage and response.events. stream_timeout_ms (default 30000) fails a stream that never closes. A recorded stream, or one in a HAR file, already gets the right stream:.

The engine runs these checks itself, and stream: and expect: survive import and export.

npx stresseur generate prints a hint on AI endpoints. Add expect: yourself, or let your coding agent do it after install-skill. To start from a browser capture, use generate --har (HAR).

On the pull request

The comment gets a Quality section when a check’s pass rate drops against the last 30 runs on main. A drop of 10 points warns. To fail the check, set fail_drop in api-tests/stresseur.yml; this needs Team:

gate:
  quality:
    pass_rate: { warn_drop: 10%, fail_drop: 25% }