Every Unit Test Passed, and Nobody Could Log In

Keep building with a coding agent and one day you have 2,000 unit tests. We didn’t wait for that day: we gave a small shop app a suite that big by asking a coding agent for at least 2,000 tests, and it wrote 2,189, all passing, covering 99% of the lines in src/. Then we made eight ordinary commits that break the app. We wrote the list down before running anything. The unit tests failed on 5 of the 8. One API flow test, recorded while using the app, failed on 6. Each side caught bugs the other missed, and one bug got past both. Then we made refactors that changed nothing a user or client sees: the unit suite went red on three of four. Below is every row. For when to write which kind of test, see unit tests vs integration tests.

The setup

The app is a small Express shop: login, categories, tags and products, backed by SQLite.

The bug list came first. Each bug is a commit a real developer makes, with a plain title like “Tidy server.js” or “Central config”. When a real developer would update a unit test in the same commit, the commit does. We committed the list, our predictions and the protocol before any test suite or result existed. The git history shows the order.

The unit suite was written without seeing the list. A coding agent, run headless in a clean copy of the shop, got one prompt: write an exhaustive unit test suite of at least 2,000 tests, covering every controller, the auth middleware, every model method, request validation and edge cases, with no duplicates and no padding. It worked for 29 minutes and wrote 2,182 tests in 16 files, next to the app’s 7 existing ones: 2,189 in all. Controller tests stub the models. Model tests run real queries against a temporary SQLite file. No test starts the server or calls it over HTTP. It does load server.js, with listen stubbed out, and checks how the app is wired.

Most of the breadth comes from tables: 508 test definitions expand into 2,189 tests over value lists (names, prices, IDs, token shapes). Roughly 45% test controllers, 37% models and the database layer, and the rest the auth middleware, the routers, server.js and the two scripts.

Line coverage is 99% of all of src/, with server.js and every file in routes/ at 100%.

Our first try used a 199-test suite written from a shorter prompt. It never loaded server.js or routes/, and it caught 2 of the 8. The bigger suite is the one we report here.

The flow side is one recorded flow with seven steps: log in, list categories, list tags, list products, create a category, create a product, update it. We added no assertions after seeing results. It ran with the published npx stresseur flow run, on a fresh database each time. The login token from the first step feeds every later step, the way multi-step API flows with auth tokens work.

Each side ran 5 times per commit. Every result was 5 out of 5, with no partial rows. Two scoreboard runs were void, and we kept both. In the first, the app had no .env file. In the second, another copy of the shop on the same machine took over the app’s port mid-run.

The scoreboard

#	Commit	What a user sees	Unit tests	Flow
B1	Tidy server.js: body parser moved below the routes	Login fails with “Internal server error”	Caught: middleware order changed	Caught: login answers 500
B2	Login response renames token to accessToken	Login never reaches the dashboard	Passed	Caught: next call answers 401
B3	/api/categories renamed to /api/category	Adding a category fails	Caught: mount path changed	Caught: create answers 404
B4	Central config: login and auth read different secrets	“Session expired. Please login again.”	Caught: login token rejected by the routers	Caught: next call answers 401
B5	New slug NOT NULL column, create not updated	Adding a category fails	Caught	Caught: create answers 500
B6	Simpler product update: no tags wipes the tags	Nothing in the browser; API clients lose tags	Caught	Passed
B7	camelCase field names in the product list	Category column shows “N/A”	Passed	Passed
B8	Create product answers 200	Nothing in the browser	Passed	Caught: 200 instead of 201
	Total		5 of 8	6 of 8

The clean master branch passed both sides 5 out of 5.

What the big suite caught

The 2,189-test suite does something the small one didn’t: it loads server.js. Express runs middleware in the order it is registered, and one test pins the exact order of the middleware stack, and others pin each router’s mount path. B1 moved the body parser below the routes, and B3 renamed a mount path, so both fail. Those tests fail on any change to the wiring, harmless or not. A team that sees them break after a tidy-up may just update them. Here, the commits didn’t touch the tests, so they count as catches.

B4 was caught by one test out of 2,189: it logs in through the auth router and sends the token to the categories router. After the commit, the login signs with one secret and the auth check reads another, so the token is rejected. The controller and middleware tests each mock the other side, and they pass.

B5 broke a real insert, and B6 failed four “keeps the existing tags” cases. The flow missed B6 because it always sends tags. It can only fail on what it sends and checks.

What it missed

B2 is the title of this post. The commit renamed the token in the login response and updated the unit tests that read it, as any developer would. All 2,189 tests pass. The web app still reads token, so nobody can log in. The flow reads the token from the login response like any client does, so its next request goes out without one and gets a 401.

B8 is the same story at a smaller scale: the commit changed the status code and the tests that check it.

B7 got past both. The product list renamed category_name to categoryName, and the tests were updated with it. The flow checks status codes and a few fields, and that field isn’t one of them. A user sees “N/A” in the category column.

Integration tests that start the app and call it over HTTP could catch B2 and B8 too. That is the category the flow belongs to. The difference is who writes it and keeps it current: the flow here was recorded while using the app, not written by hand.

The cost: red on harmless refactors

A suite that pins how the app is wired catches wiring bugs, and it also fails when the wiring changes for a good reason. We measured that too. Before running anything, we wrote down five refactors we expected to be harmless, with no test edits, and ran both sides 5 times on each.

#	Refactor	Unit tests	Flow
H1	Body parsers swapped (form parser before JSON)	Red (1 test: middleware order)	Green
H2	Internal database helper renamed	Green	Green
H3	Product option routes moved to their own router file	Red (4 tests: router contents)	Green
H4	Category “not found” message reworded	Red (9 tests: exact message)	Green
H5	Routers mounted in alphabetical order	Red (2 tests: mount positions)	Green

H4 turned out not to be harmless in the sense that matters: it changes a response body that API clients receive, so a unit test that asserts the message is arguably doing its job. Leave it out, and the suite went red on three of four refactors that changed nothing a user or client sees (H1, H3, H5), every run. Those three are the same kind of test that caught B1 and B3: tests that pin the middleware order, the mount order and each router’s contents. The catches and the false alarms come from one design choice. Each fix is a one-line test update, so this is friction, not a disaster. But it teaches a team to update failing tests on sight, and that habit is how B2 and B8 got through: those commits updated their tests too.

Don’t read the flow column as proof that the flow is sound. It never calls the option routes (H3) or asks for a missing category (H4), so it could not have failed there. We checked the option routes by hand on H3, and they behave the same, auth included.

How long each one takes

We timed 50 alternating runs of each on the clean branch, on a laptop and in a 2-CPU container sized like a CI runner. The flow needs the app running, so we also timed app start to the first answer, 20 times (same app code as our first run).

Medians: on the laptop, the unit suite took 5.1 s and the flow 0.37 s, and the app started in 0.12 s. In the container, the unit suite took 17.6 s and the flow 0.25 s, and the app started in 0.09 s.

We don’t claim a speed win. In the container the unit suite is slow because node --test runs one process per test file and, on 2 CPUs, one file at a time. One container run of the unit suite hung and never finished. We stopped it, threw the batch away and ran the 50 again with a time limit. The coding agent had flagged the likely cause in its own report: in this app, db.ready resolves before the tables exist, and a failed database open never settles. Both numbers still fit in every pull request.

What to do

Keep your unit tests. They caught B6, which the flow could not, and when they fail they point at the line. A big suite that loads your wiring catches more than a small one. It also goes red on harmless refactors, so expect to update tests when you refactor, and read each update before you accept it.

Add an API flow test and run it on every pull request. It goes through the real routes, the real login and the real database, and it uses the response the way a client does. With Stresseur, npx stresseur record turns how you use the app in development into flows under api-tests/, and npx stresseur init-ci writes the GitHub Actions workflow that runs them. See API integration tests in Stresseur, the Stresseur quickstart guide and CI setup for pull requests.

Then read your flow the way B7 reads it. A flow only fails on what it checks. If a field matters to your users, assert on it.