Runs of the frozen PR review eval suite, published as measured: what was tested, what was found, what missed, and the configuration that produced each.
The runs
| Run | What was tested | Corpus | Recall | Fabricated | Frozen authz | Configuration | Date |
|---|---|---|---|---|---|---|---|
| Run 1 | PR review, general scope, three passes per planted pull request | 57 planted pull requests: 27 flaw plants under the general class list, 12 frozen authorization plants, 18 benign look-alikes | 26 of 27 (96.3%) | 0 | 10 of 12 | The production PR review lane, reasoning effort low | 25 September 2026 |
| Run 2 | PR review, general scope, three passes per planted pull request | 57 planted pull requests: 27 flaw plants under the general class list, 12 frozen authorization plants, 18 benign look-alikes | 25 of 27 (92.6%) | 0 | 11 of 12 | The production PR review lane, reasoning effort low | 25 September 2026 |
Each recall number is the result of that named run on that corpus. It is not a product accuracy rate, and results on your code depend on your code. New runs of the customer setup get appended to this table as they happen, worse results included.
Fabricated counts a finding published on a benign plant. On the founding runs of 25 September 2026, all 18 benign look-alikes stayed silent. The internal gate behind a configuration change sets a recall bar of 83.3% on the general corpus and a bar of 10 of 12 on the frozen authorization slice, with zero fabricated; a published finding that does not match a planted line fails a run too.
Method
The scan-side record: the transparency report.
Honest limits
The corpus is small: single-flaw planted pull requests of a few thousand tokens each, so these numbers rank configurations against each other; they are not a forecast of results on your pull requests. The founding runs of 25 September 2026 are two runs of this configuration; the wider record shows run-to-run swings of up to four plants on other lanes, so read each number as that run's result. The thinnest class is path traversal: the founding runs found one of two traversal plants each, and the second is a consistent miss. The judge is strict by design: a finding that names the right bug but does not quote the planted line counts as a miss, and one plant in the corpus carried a second real bug that produced a true extra finding in a replication run; the record keeps both facts. Zero fabricated over 18 benign plants is the precision record of the founding runs; it says nothing about precision on unseen code.
Scope
PR review looks for access control, business logic, SQL and command injection, XSS, SSRF, path traversal, unsafe deserialization, token and session flaws, and LLM-integration risks. Whole-repo scans read authorization, IDOR, and business-logic access control only, and stay access-only by design. These receipts cover PR review; the scan side keeps its own frozen record.