Two Slower PRs: What Caught Them

We opened two ordinary pull requests on a small shop app. One raised the bcrypt cost from 10 to 13, a reasonable security change (OWASP sets 10 as the minimum). It made login about 8 times slower. The other added a product count to each tag. It more than doubled the time of the product list. The 199-test unit suite passed on both. Your coding agent reviewed each pull request three times. It named the login cost every time and approved anyway, and it asked for changes on the tag count every time. So the reviewer did notice. What it can’t do is measure every pull request the same way and block the merge on a number. That is the job of a check.

What we ran

The app is the same small Express shop from our unit test experiment, seeded with 200 products and 20 tags so list endpoints do real work. We wrote down both pull requests, and 10 no-change pull requests for noise control, before running anything.

• PR1, “Raise bcrypt cost to 13”: one line in the user model, bcrypt.hash(password, 10) to 13.
• PR2, “Show how many products use each tag”: each tag on a product gets a product_count, read with one COUNT query per tag per product, a classic N + 1 query.

For timing we used the published CLI: npx stresseur ci runs the flow once to warm up, then 12 measured times, and reports each step. We ran it 3 times per branch, each on a fresh seeded database, on a laptop and in a 2-CPU container sized like a CI runner.

One thing we did not run: the Stresseur pull request check. The demo repo isn’t connected to Stresseur Cloud yet, so no check or comment was posted on these pull requests. Every number below is from the local runs.

What the timings showed

	Step	Before	After	Change
PR1, laptop	Log in	61.0 ms	468.5 ms	+668%
PR1, 2-CPU container	Log in	50.2 ms	393.9 ms	+685%
PR2, laptop	List products	15.6 ms	32.7 ms	+110%
PR2, 2-CPU container	List products	17.7 ms	41.2 ms	+133%

Medians, all three runs agreeing on each row. The whole flow went from 82.9 ms to 490.7 ms on the laptop for PR1 and to 100.0 ms for PR2. In the container it went from 81.1 ms to 425.6 ms and 103.7 ms. The flow’s assertions passed on every run. No step failed. It only got slower.

What the unit tests saw

The 199-test suite passed on PR1, on PR2 and on all 10 no-change pull requests. That is the right result. A unit test checks what the code returns, and neither change broke what these tests check.

One correction from our notes: the first version of PR2 failed 4 unit tests, because those tests compare tag objects exactly and PR2 adds a field. Its author would update those assertions in the same pull request, so PR2 now does. The tests failed on the new field, not on time.

What your coding agent’s review saw

We asked for a review three times per pull request with the same prompt: “Review this pull request as a senior engineer. Approve or request changes.”

PR1: approved 3 of 3, and all 3 named the login cost. The reviewer timed a hash itself, about 59 ms at cost 10 and 466 ms at cost 13. It pointed out that the login route has no rate limit, so each attempt now costs about half a second of CPU. One review called the cost “modest”. Another said it had not run the API flow test: “Each logs in once, so the extra ~0.4 s should not matter, but I did not confirm it.” It also found a gap we never planned: existing users keep their cost-10 hash until something rehashes it.

PR2: requested changes 3 of 3. Every review flagged the extra query per tag per product on the product list. It also found that the count is wrong after a product is deleted, and that the count lands on the wrong endpoint for the feature the commit describes. Those are two correctness bugs no timing run would find.

That is a good reviewer. It noticed both slowdowns, and on PR1 it made a fair call: half a second per login may be fine for an admin panel.

Where a check fits

A review is a judgment made once, when someone asks for it. On PR1 the judgment was “acceptable”, from a hash timed on the reviewer’s machine. Nobody measured the login endpoint, and the next pull request that touches login starts from zero.

A check does the same thing on every push. In CI, Stresseur runs each flow 13 times (1 warm-up, 12 measured) and uploads the time of every step. It compares the median with the last 30 comparable runs on the default branch. A flow is flagged when it is at least 10% slower and outside its normal run-to-run spread. The pull request gets one check and one comment, updated on each push, with the flow time before and after and the steps that took the time.

Here is what the default gate would make of the numbers above. It warns above +10% and fails above +30%:

• PR1’s whole flow went up 492% on the laptop. That is past the fail line, and the comment would point at the login step.
• PR2’s whole flow went up 21% on the laptop and 28% in the container. That is a warning, with the product list as the step that took the time.

Two limits to know. On GitHub-hosted runners the default gate only warns, and a fail_above you write in api-tests/stresseur.yml always applies. Free plans warn only. A check also needs a baseline: with fewer than 10 runs on the default branch, it only warns.

False alarms matter as much as catches. We set up 10 no-change pull requests to test that. So far only the unit suite has run on them; the check results come once the demo repo is connected.

What to do

Keep the review. It found two bugs in PR2 that timing never would, and it saw the login cost.

Add a number next to it. Run npx stresseur ci before and after a change you suspect, and you get per-step medians like the table above. Then run it on every pull request with npx stresseur init-ci, and write the line you care about in api-tests/stresseur.yml:

gate:
  flow_time:
    warn_above: 10%
    fail_above: 30%

The reviewer decides if a slower login is worth it. The check makes sure someone decides, with the number in front of them. See API tests on every pull request, CI setup for pull requests and gates in stresseur.yml.