Writing · Evidence

It built itself. Then a human found what the tests could not.

The absence this makes loud: a suite that is green about the wrong thing.

The dogfooding account, including the number that matters most — how many of the defects found by hand the test suite had caught.

2026-09-08 · 5 min read

The repo is its own case study: the specs were grilled in Graphban, decomposed into items in Graphban, built by a fleet of agents claiming through Graphban, and reviewed by other agents that the server would not let sign off their own work.

48
days old
first commit 2026-07-21
1,468
commits
808 non-merge
57
MCP tools
as of 2026-09-07
4,200
tests green
3,471 backend · 729 frontend
0
defects the suite caught
of those found by hand

These figures were last synced from the source repo on an earlier date and may be behind. A stale number that reads as current is the thing this whole page is about, so the page says so rather than hoping someone remembers to sync it.

A green bar representing 1,748 passing tests, with nine defect markers below it that the suite did not catch. 1,748 tests passing · 2026-08-13 GRPH-377 ×4 · 378 · 379 · 395 · 396 · 376 found by a human driving four terminals · found by the suite: 0
The green bar is real, and it is not evidence. Nine defects, documented with ticket ids, every one invisible to a suite that was green throughout.

What the walk found

Eight steps run against a real stack with real agents. The walk's own summary: "Five defects found, four fixed, one filed." Its defect table lists nine rows across six ticket ids — a count that drifted inside the document written to catch drift, which is either embarrassing or exactly the point.

TicketThe absence
GRPH-377The role gate was unreachable. update_item never advertised agent_id, so the ceiling had never gated anything in production — leaving a hole under the self-review ban.
GRPH-377Reviewers and planners died 150s after registering. heartbeat was gated to workers — refusing the only call that keeps an agent on the roster.
GRPH-378A bounce required a reason and discarded it. After a real bounce the string appeared in no row of any table. The author got the item back with nothing to act on — the exact failure the requirement existed to prevent.
GRPH-379The pin was invisible. A refused claim returned {"claimed": false, "item": null} — byte-identical to an empty backlog. A worker that should idle and retry concludes the project is finished.
GRPH-395A review claim never expires. A reviewer that dies strands the item in review forever, looking like ordinary queued work.
GRPH-396A dead agent's branch is never flagged. The flag is written only inside quarantine — reachable only by an agent that is demonstrably alive. The agent that crashes holding a branch is recorded nowhere, and that is the common case.
GRPH-376The self-review ban is unprovable after the fact. Sign-off clears the claim, so every done item reads built_by: -. Enforcement is fine; the audit trail is not. Filed, not fixed.

Predictions, scored

Stated in advance so the walk would test claims rather than confirm them. One of four held.

PredictionOutcome
Vendor strings never tested against real clientswrong real clients report xai and cursor
host is self-reported and nothing reports itwrong reported, and seats made it moot
wait_seconds has never parked against a real proxyright 504 at exactly 60s
The directive downlink has never reached a real clientwrong arrives, once, on a heartbeat

← All writing

What's new

Every entry is work that merged. The full history is on GitHub.