A QA agent that remembers
what broke since yesterday.
I test AI systems for a living, so I know what most AI "QA agents" actually do: they re-audit the whole repo every run, re-report the same findings until you stop reading, and sign off with a cheerful LGTM. Verdict is the tester I wanted instead. It keeps a baseline, reports only what changed, and is structurally unable to edit the code it judges. It is MIT-licensed, and it publishes its misses next to its hits.
A reviewer that never says no is not a reviewer.
Point a general-purpose AI agent at a codebase and ask it to test, and you get an audit written from scratch every single time. It cannot tell a flaky test from a broken one, or a stale expectation from a real regression — so everything red becomes a "failure", everything green becomes approval, and the whole thing ends in an emoji. By the third run you are skimming it.
That matters more once agents are writing the code. A loop where the same agent implements, tests, and then declares itself finished has no independent gate in it at all — it is grading its own homework. I needed a tester that argues back, remembers what it already told me, and is willing to say it could not check something.
Read-only by construction. Skeptical by contract.
Two properties do most of the work: the tester cannot touch the code it judges, and it remembers every run that came before.
It cannot fix what it finds
The agent ships with no Edit tool at all. Its Write access is scoped to the QA directory — reports, state, nothing else — and a PreToolUse hook backstops that scope if the contract is ever ignored. It specifies tests rather than writing them: the acceptance criterion, the precise assertion, the fixture. The README is candid that the hook is a hard guarantee only in strict mode; a QA tool should not oversell its own controls.
A repeat run is a delta, not a re-audit
The first action of every run is to read the stored baseline; the last is to write it back. Findings get a stable ID hashed from path, rule, and normalized message, so an ID survives the line numbers moving underneath it. That turns each run into NEW, REGRESSED, STILL_OPEN, and RESOLVED — regressions ranked first, and every open finding carrying its age in days, because age is the pressure. Every line a finding cites is hashed as well, so the next run knows which evidence the code moved under it and re-verifies exactly that — and when nothing a finding cites has moved, a night carries the standing verdict without calling a model at all.
The judgment is a contract. The memory is an API.
A slash command starts a run, the agent works read-only against your repo, and everything it concluded lands in a state file that anything else can read.
Delta report
NEW · REGRESSED · STILL_OPEN · RESOLVED
One verdict
pass · pass with risks · blocked · fail
Quarantine ledger
every flake carries an expiry date
MCP server
10 read-only tools · verdict-gate turns the state into a CI exit code
The rules that make it disagree with you.
Most of these exist to close a specific way an eager agent talks itself into a green result.
Quarantine, but only with an expiry
A flaky test is confirmed by re-running it, then quarantined with a date attached — excluded from the verdict, never from the report, and force-re-evaluated when the date passes. The inverse is a finding in its own right: a test skipped "temporarily" with no expiry is a graveyard entry, and gets flagged as one. A loop that can skip its way to green will.
"The test is just outdated" needs a receipt
Of the five ways a failure gets classified, one is the escape hatch — deciding the code is right and the test is stale. That is the classification most likely to be wrong in your favour, so it alone requires a citation proving the behaviour change was intended: a commit, a changelog, a requirement. Without one it is a real defect, and gets filed as a real defect.
The gate of a fix loop, never its actor
Verdict finds and judges; the pen stays with whoever is implementing. This is the whole reason it has no Edit tool. An agent that patches the code and then re-runs its own judgement on the patch has quietly removed the only independent step in the loop.
The memory is public API; the pen is not
Ten MCP tools expose the verdict, the findings, the quarantine ledger, the questions it parked for a person, and the run history — every one of them marked read-only. So CI can gate a merge on it, another agent can read it, an editor can surface it. The maintainer keeps two pens the tester cannot hold: one accepts a finding's risk with a citation, the other answers a question the tester parked.
Four verdicts, and "blocked" is one of them
A run ends on exactly one of pass, pass with risks, blocked, or fail. Blocked means it could not verify — a missing tool, an unsafe environment — and it is a legitimate outcome rather than a failure to report one. An open blocker forces a fail. A pass without a stated list of what was not covered is treated as incomplete.
It ships with an eval it can fail — and publishes the misses
Seven scored fixtures: seeded defects, a delta run against an authored history, a TypeScript twin, root-cause analysis with a decoy suspect, a spec review before any code exists, AI-generated slop, and an adversarial repo whose test script prints ALL TESTS PASSED while exiting 1. Then a key nobody here wrote: 40 SWE-bench Verified issues, the maintainers' fix located in 36 of 37 scored runs — and a control arm, published beside it, in which plain Claude Code did just as well in a twelfth of the time. A QA tool that hides its own misses has no business grading yours.
Everything here is readable in the repo.
- Public and MIT, 73 tagged releases in its first month: a 797-line agent contract, a fact harness and exit-code gate, and 1,338 tests on a Linux + Windows CI matrix.
- The honesty trap it once caught 1 run in 3 — a conftest that silently skips every test — is now caught every time, and not by rewording the prompt: the harness counts the skipped tests itself, so the model is handed a fact instead of asked for one.
- In the seeded eval, a test skipped since May with the note "flaky?" was correctly called not-flaky — it was the exact test that would have caught the seeded defect.
- Pointed at strangers' repositories, it found a rotate_file in boltons that, at keep=1, deletes the file it rotates, and a FilePerms bug that had lived 4,595 days under 625 green tests — both reported upstream after an independent reproduction.
- Turned on itself — 78 findings in its first 15 runs — it filed the one that hardened its own anti-fabrication gate: runs now sign the history with a hash chain a copied state cannot reproduce.
- Install is one line: /plugin marketplace add ArtJack/verdict — or npx skills add ArtJack/verdict in any other coding agent.
Refusing was the feature
The first fully unattended run refused to execute the test suite at all: it found a live .env file sitting in the checkout and would not run against it. It blocked the gates, said plainly that it had done so, and still delivered a delta report from read-only data. Then it found a real leaked API token, untracked, in the same repo. A tool that had been optimizing for a green tick would have run the suite and reported success.
What's next
An independent re-run of the eval by someone who is not me — that is still the number I would most like to replace. And a key that scores what Verdict is for: the SWE-bench control showed that locating one defect from one issue report is something plain Claude already does well, so the next external measurement has to see memory across runs, fix verification, and the gate.
Want a second set of eyes that isn't the one that wrote the code?
Verdict is free and open source — install it and point it at your repo. If you would rather have the judgment than the tool, independent QA on software you or another team built is one of the things I do; you get a written report of what breaks and why.