A QA agent that remembers
what broke since yesterday.
I test AI systems for a living, so I know what most AI "QA agents" actually do: they re-audit the whole repo every run, re-report the same findings until you stop reading, and sign off with a cheerful LGTM. Verdict is the tester I wanted instead. It keeps a baseline, reports only what changed, and is structurally unable to edit the code it judges. It is MIT-licensed and small enough to read in a sitting.
A reviewer that never says no is not a reviewer.
Point a general-purpose AI agent at a codebase and ask it to test, and you get an audit written from scratch every single time. It cannot tell a flaky test from a broken one, or a stale expectation from a real regression — so everything red becomes a "failure", everything green becomes approval, and the whole thing ends in an emoji. By the third run you are skimming it.
That matters more once agents are writing the code. A loop where the same agent implements, tests, and then declares itself finished has no independent gate in it at all — it is grading its own homework. I needed a tester that argues back, remembers what it already told me, and is willing to say it could not check something.
Read-only by construction. Skeptical by contract.
Two properties do most of the work: the tester cannot touch the code it judges, and it remembers every run that came before.
It cannot fix what it finds
The agent ships with no Edit tool at all. Its Write access is scoped to the QA directory — reports, state, nothing else — and a PreToolUse hook backstops that scope if the contract is ever ignored. It specifies tests rather than writing them: the acceptance criterion, the precise assertion, the fixture. The README is candid that the hook is a hard guarantee only in strict mode; a QA tool should not oversell its own controls.
A repeat run is a delta, not a re-audit
The first action of every run is to read the stored baseline; the last is to write it back. Findings get a stable ID hashed from path, rule, and normalized message, so an ID survives the line numbers moving underneath it. That turns each run into NEW, REGRESSED, STILL_OPEN, and RESOLVED — regressions ranked first, and every open finding carrying its age in days, because age is the pressure.
The judgment is a contract. The memory is an API.
A slash command starts a run, the agent works read-only against your repo, and everything it concluded lands in a state file that anything else can read.
Delta report
NEW · REGRESSED · STILL_OPEN · RESOLVED
One verdict
pass · pass with risks · blocked · fail
Quarantine ledger
every flake carries an expiry date
MCP server
9 read-only tools · verdict-gate turns the state into a CI exit code
The rules that make it disagree with you.
Most of these exist to close a specific way an eager agent talks itself into a green result.
Quarantine, but only with an expiry
A flaky test is confirmed by re-running it, then quarantined with a date attached — excluded from the verdict, never from the report, and force-re-evaluated when the date passes. The inverse is a finding in its own right: a test skipped "temporarily" with no expiry is a graveyard entry, and gets flagged as one. A loop that can skip its way to green will.
"The test is just outdated" needs a receipt
Of the five ways a failure gets classified, one is the escape hatch — deciding the code is right and the test is stale. That is the classification most likely to be wrong in your favour, so it alone requires a citation proving the behaviour change was intended: a commit, a changelog, a requirement. Without one it is a real defect, and gets filed as a real defect.
The gate of a fix loop, never its actor
Verdict finds and judges; the pen stays with whoever is implementing. This is the whole reason it has no Edit tool. An agent that patches the code and then re-runs its own judgement on the patch has quietly removed the only independent step in the loop.
The memory is public API; the pen is not
Nine MCP tools expose the verdict, the findings, the quarantine ledger, and the run history — every one of them marked read-only. So CI can gate a merge on it, another agent can read it, an editor can surface it. Nothing outside the agent can write to it.
Four verdicts, and "blocked" is one of them
A run ends on exactly one of pass, pass with risks, blocked, or fail. Blocked means it could not verify — a missing tool, an unsafe environment — and it is a legitimate outcome rather than a failure to report one. An open blocker forces a fail. A pass without a stated list of what was not covered is treated as incomplete.
It ships with an eval it can fail — and publishes the misses
Six scored fixtures: seeded defects, a TypeScript twin, root-cause analysis with a decoy suspect, a spec review before any code exists, AI-generated slop, and an adversarial repo whose test script prints ALL TESTS PASSED while exiting 1. The results table carries the failures alongside the 8/8s: measured at n=3, one honesty trap — a conftest that silently skips every test — is caught 1 run in 3. That number is in the repo because a QA tool that hides its own misses has no business grading yours.
Everything here is readable in the repo.
- Public and MIT, 43 releases: a 754-line agent contract, a fact harness and exit-code gate, and 419 tests on a Linux + Windows CI matrix.
- Six scored eval fixtures with published results — including the misses: one adversarial trap is caught 1 run in 3, measured at n=3 and recorded in the repo next to the 8/8s.
- In the seeded eval, a test skipped since May with the note "flaky?" was correctly called not-flaky — it was the exact test that would have caught the seeded defect.
- Turned on itself, it filed the finding that hardened its own anti-fabrication gate: runs now sign the history with a hash chain a copied state cannot reproduce.
- Install is one line: /plugin marketplace add ArtJack/verdict
Refusing was the feature
The first fully unattended run refused to execute the test suite at all: it found a live .env file sitting in the checkout and would not run against it. It blocked the gates, said plainly that it had done so, and still delivered a delta report from read-only data. Then it found a real leaked API token, untracked, in the same repo. A tool that had been optimizing for a green tick would have run the suite and reported success.
What's next
An independent re-run of the eval by someone who is not me — that is still the number I would most like to replace. Nearer term: fixing the 1-in-3 trap, which costs a prompt change plus enough re-runs to prove the fix moved that row and nothing else.
Want a second set of eyes that isn't the one that wrote the code?
Verdict is free and open source — install it and point it at your repo. If you would rather have the judgment than the tool, independent QA on software you or another team built is one of the things I do; you get a written report of what breaks and why.