Most AI 'QA agents' re-audit the whole repo every run, re-report the same findings until you stop reading, call flaky and stale tests alike 'failures', and sign off with an LGTM. And once an agent is writing the code, a loop where it also tests its own work has no independent gate left in it.
What I builtA QA agent with a stored baseline, so a repeat run is a delta — NEW, REGRESSED, STILL_OPEN, RESOLVED — with regressions ranked first and every open finding carrying its age. It has no Edit tool by design, quarantines flaky tests only with an expiry date attached, requires a citation before it will write a failure off as a stale expectation, and closes on one of four verdicts where 'blocked' is a legitimate answer.
Built withZero-dependency Claude Code plugin, 10 slash commands behind one front door (/verdict:run picks baseline, delta, or a scoped review itself), a Python MCP server exposing 9 read-only tools, a fact harness that measures counts and diffs so the model contributes judgment only, an exit-code release gate for CI, a 24-technique test-design catalog, 5 report templates, and six hooks — write-scope, Bash-scope, and a Stop hook that blocks a run which tried to hand-write its own state. MIT.
TestedShips with its own eval: six scored fixtures — seeded defects, a TypeScript twin, root-cause with a decoy, a spec review with no code, AI-generated slop, and an adversarial repo whose suite prints ALL TESTS PASSED while exiting 1. Published results include the misses: measured at n=3, one honesty trap is caught 1 run in 3, and the repo says so next to the 8/8s. 419 tests and a scorer-regression corpus run in plain CI with no model.
ResultPublic and MIT, 43 releases. Runs unattended nightly against a production codebase — its first fully unattended run refused to execute the suite because a live .env sat in the checkout, said so, and still delivered a delta report, then found a leaked API token untracked in the repo. Turned on itself, it filed the defect that made its own anti-fabrication check imitation-proof: every run now signs the run history with a hash chain a copied state cannot reproduce.