PROOF verification benchmark · a living log

We let a second agent try to break our own fixes.

Coretexa is the rule that the agent which does the work can't close it. This is the receipt: every time we run the loop against a real open-source bug, an independent verifier tries to refute the fix before it ships. Here is what it caught.

last updated 2026-07-27 · ● live — appended as runs happen

Fix runs
19 + 3 reported
Bot PRs refuted
5 / 1 fixed
Verifier catches
11 of 12 written up
First-pass-correct
1
Withdrawn by us
1 duplicate
Merged
5 / 11 in review

Five merges landed: a Bazel maintainer merged the buildtools fix into bazelbuild:main (Google/Bazel core tooling); the FastMCP auth fix merged the same evening it posted; and — the marquee — a Google maintainer merged the gson fix into google/gson after running it through Google’s full internal test suite first (a library with 1B+ downloads). Ecosystems: TypeScript · Python · Rust · Go · Java — including GitHub’s own gh CLI and the MCP reference SDK. Run 6 is the first first-pass-correct run: the doer nailed it and the verifier said so. We report that honestly rather than claim a perfect catch rate — a catch rate you can’t distinguish from a clean pass isn’t worth anything. The fourth merge landed 07-25: a Sollumz maintainer merged the szio fix six hours after a user filed the bug — and it was the second version of that fix, because the verifier killed the first one for silently corrupting data while all 291 tests passed. Also in flight: superfile (a 1.36GB memory blowup traced to a mis-parsed video header) and QuantEcon.py. The fifth merge landed 07-27: git-machete, after four rounds of review across nine commits — the maintainer invited us as a collaborator partway through. On croniter, a Pallets-eco maintainer diffed 8,012 schedule expansions to characterise the blast radius himself before approving in principle — the most rigorous review any of this work has received, and worth more to us than the merge. Two entries below are not wins: a pydantic PR closed by policy because we read the PR template and not the contributing guide, and a cachetools fix we withdrew as a duplicate of an open PR our own check missed. Runs 1–12 carry the full catch write-ups; 13–19 are logged in brief. Meanwhile the same hostile-verification discipline opened a second front: refuting other agents’ green-CI pull requests (see REFUTE, below).

A passing test suite lied — and the verifier caught it.

Fixing a panic in rust-url (the URL crate nearly every Rust program depends on), the first fix made the project's entire Web Platform Test suite pass green. It was still wrong: it silently dropped Windows drive letters, turning file:///a:/.. into file:///. The bundled test data had no case for it, so the green was a false green.

The independent verifier didn't trust the green. It ran a differential sweep against the WHATWG reference, caught the regression, and rejected the fix. The doer narrowed it; the second pass verified clean. That is the entire point of Coretexa — the actor that did the work was not allowed to close it.

Honest postscript (07-24): Servo declined the PR — “the Servo project does not currently allow AI contributions.” A policy call, theirs to make; we logged it as rejected-policy, added servo/* to our do-not-post list, and the catch above stands on its own merits. And the pattern repeated: on 07-25 the very same false-green failure mode was caught again, in szio — a fix that passed all 291 tests and was still silently corrupting data (first entry in the log below).

LOG the run history
2026-07-27 pydantic/pydantic #13507 Python closed · policy

Every constraint chained after ge/lt/le vanished from the published JSON schema

In pydantic’s experimental pipeline API, validate_as(int).ge(0).multiple_of(2) emitted a JSON schema carrying only the ge. The dropped constraint still ran at validation time — so the model rejected odd numbers while the schema it published said they were fine. Every consumer downstream of that schema — a generated client, an OpenAPI document, a form builder — would have disagreed with the server, and nothing would have logged an error anywhere.

Closed by policy, and the mistake was ours. A bot closed the pull request within minutes: pydantic requires a contributor to be assigned to the referenced issue before opening one. That rule is stated plainly in docs/contributing.md — a file that was sitting in our own checkout, unread. We read the PR template and not the contributing guide. The issue stands open on its merits; the PR is logged here as a process failure, not a technical one.

2026-07-26 knadh/koanf #434 Go PR open

Int64 returned the most negative number for the largest positive one

koanf.Int64 falls through to a float round-trip for any non-integer input and finishes with int64(f). Go leaves that conversion implementation-defined when the value is out of range; on amd64 and arm64 the hardware hands back the “integer indefinite” value, which is math.MinInt64. JSON is what makes it reachable from an ordinary config file, because encoding/json decodes every number to a float64 and float64(math.MaxInt64) rounds up past the int64 range. So {"n": 9223372036854775807} — the largest legal int64 — read back as -9223372036854775808.

The guard that does not work: the obvious bound, f > math.MaxInt64, is false for exactly the values that overflow. math.MaxInt64 is an untyped constant, and it converts to the same float64 that the overflowing value rounds to, so the comparison never fires — a guard that looks right, compiles, and silently lets every bad value through. The fix bounds on float64(1 << 63) instead. float64(math.MinInt64) is exactly representable, so it is correct on the low side; NaN compares false against everything and needs its own test.
2026-07-25 gtk-rs/gtk4-rs #2345 Rust PR open

A refactor moved eight methods onto a trait no widget subclass can reach

Since 0.11.4, a gtk4-rs widget subclass with a custom class struct could no longer call add_binding_action from class_init() — nor add_shortcut, install_property_action, query_action, activate_signal, layout_manager_type, css_name or accessible_role. A commit titled “allow calling Class methods on any widget” moved all eight off WidgetClassExt (bounded on ClassStruct) and onto a new WidgetClassManualExt implemented for glib::Class<T>. Those two bounds are disjoint, so the methods became unreachable from precisely the place the documentation tells you to call them.

The issue reported one method. All eight were gone. The affected version window is not recoverable from the tags — git tag --contains on the offending commit returns nothing — so it was established by checking out released tags one at a time and compiling against each.

2026-07-25 Sollumz/szio #1217 Python ✓ MERGED ✓ verifier caught

All 291 tests passed — and the fix was still silently corrupting a matrix shape the suite never checks

Sollumz (the standard GTA V Blender toolchain) crashed importing any model whose CompositeTransform used tabs: the matrix parser in szio split rows on literal 3-space runs and values on single spaces, so tab-delimited files died with ValueError: could not convert string to float: '0\t\t\t\t\t0'. Scouted 5 hours after a user filed it, reproduced byte-identical, root-caused, fixed with tests — PR posted 26 hours after the issue opened, race-clean the whole way.

Catch — the false green, again: the doer’s first fix (flat whitespace tokenization) passed the entire 291-test suite green. The adversarial verifier corpus-diffed old-vs-new over the repo’s own fixtures and killed it: CodeWalker writes fragment bound matrices as 4 rows × 3 columns, and the flat fill silently smeared orientation and translation — demonstrated end-to-end through the public Fragment API, on files the project itself ships. No test asserted those values, so CI stayed green over live data corruption. v2 preserves per-row structure; both verifiers re-ran clean in pristine trees. The run also surfaced two latent bugs nobody had filed (a wrong return class, and a crash under the standalone numpy backend).

Merged 07-25, six hours after the bug was filed: a Sollumz maintainer merged it into szio:main (commit 5aa3d16), auto-closing the Sollumz issue — the benchmark’s fourth merge. Worth stating plainly: the version that merged is the second one. The first passed every test and would have shipped silent data corruption.

2026-07-25 QuantEcon/QuantEcon.py #870 Python PR open ✓ verifier caught

A download utility saved HTTP error pages under the requested filename — and reported success

fetch_nb_dependencies, exported at the top level of QuantEcon.py (the standard computational-economics library), wrote whatever the server returned straight to disk and appended True. Ask for a file that isn’t there and you get a 314 KB HTML error page saved as your-data.csv, reported as fetched — the failure resurfaces later as a parse error somewhere unrelated. Four defects in ~30 lines: no raise_for_status(), no timeout, a file handle shadowing the loop variable, and type(files) == list sending tuples into the wrong branch.

Catch — the test gap was ours: the adversarial verifier mutation-tested our own new tests and found one that couldn’t fail: reverting the default timeout to None (reintroducing the unbounded-hang bug) left the suite green, because we only asserted the explicit timeout path. Fixed by pinning the default. It also surfaced a consequence the issue never mentioned — with overwrite=True the old code destroyed a good local file, replacing real data with the error page and still returning success. That is silent data loss, now disclosed in the PR.

A quieter echo of the same theme: the two pre-existing tests for this function passed by saving an error page in any environment without network access — the bug passing its own test, in a third repo this week.

2026-07-25 yorukot/superfile #1550 Go PR open ✓ verifier caught

19 GB of RAM eaten by a file manager — because Go’s PE parser accepted a video as a COFF object

A user reported superfile filling RAM and swap until the kernel killed it, just from scrolling past large videos, and blamed the exif reader. It wasn’t exif. GetBinaryArchitecture runs on every file, and Go’s debug/pe accepts files without an MZ header as raw COFF objects — so an MP4’s leading bytes get read as section and symbol counts, and the parser allocates against garbage offsets inside a 2 GB file. One goroutine fires per cursor move, so a sweep multiplies it. The fix gates all three debug/* parsers behind a 4-byte magic check. Measured over 200 concurrent fetches: +1,357 MB → +0.6 MB, 18s → 0.01s.

Catch: the verifier ran ~40 fixtures — real ELF and .so, cross-compiled PE, thin/byte-swapped/fat Mach-O, a clang-produced COFF object, permission-denied files, fifos — and found one honest parity gap: MZ-less RISC-V COFF objects would have lost their architecture label. Three constants fixed it. It also measured the residual we chose to keep (a 2-byte magic collision still reaches the parser, ~5 in 65536 on random data), which is disclosed in the PR rather than quietly shipped.

Posted ~3 hours after the report. The repo’s own AI reviewer looked at the change and returned “no actionable comments” — on a fix for a bug that AI review had never caught in the first place.

2026-07-21 VirtusLab/git-machete #1754 Python ✓ MERGED ✓ verifier caught

discover/status showed a different branch tree in every worktree

git-machete inferred branch structure from the HEAD reflog — which is per-worktree in git — so the same repo produced different discovered trees depending on which worktree you ran it from. The fix reads reflogs across all worktree HEADs. Found the same way as the szio bug: a fresh user-reported issue, hours old, race-checked immediately.

Catch: the fix replaced a git subprocess call with direct file reads — and the hostile verifier caught that this silently dropped the subprocess’s implicit strict-UTF-8 decode. On a non-UTF-8-locale environment (minimal containers, LC_ALL=C CI), one accented committer name in a reflog would have crashed the command — a regression the doer’s green tests never touched. Fixed with an explicit utf-8/surrogateescape read that is strictly more robust than the old path.

Merged 07-27: nine commits across four rounds of review — the longest engagement on the board, and the one that produced the most correction. The maintainer issued a collaborator invite partway through. Merge five.

2026-07-20 pallets-eco/croniter #246 Python PR open ● maintainer approved in principle

59/15 * * * * fired four times an hour — and never at :59

When a step expression’s start equals the field maximum, croniter expanded {start}/{step} to the entire field. A job scheduled 59/15 * * * * ran at :00, :15, :30 and :45 — four times an hour, at none of which the user asked for. The same held in every field at its maximum: 23/6 in hours, DEC/3 in months, SAT/2 in day-of-week, 31/5 in day-of-month. The cause is a normalisation artefact read as user intent: {start}/{step} is rewritten to {start}-{max}/{step}, which at the maximum becomes {max}-{max}/{step} and falls into the branch meant for a user-written equal range like Jan-Jan, which croniter deliberately expands to the whole cycle.

Reviewed by a Pallets-eco maintainer. Jarek Potiuk (potiuk) did not take the claim on trust — he characterised the blast radius himself, diffing .expanded across 8,012 cases (4,006 generated expressions × two modes). 1,996 differ, every one of them containing a field-max X/step token, and none change error status. His verdict: “approving in principle.” He also caught two formatting violations the project’s own CI structurally cannot — tox -e fmt runs ruff format, not --check. Exactly the failure mode this whole benchmark is about, found in the review of our own patch.
2026-07-20 python-poetry/tomlkit #557 Python PR open

A sub-table under a dotted key escaped its parent and reappeared at the document root

Given a document containing [t] with a.b = 1, assigning doc["t"]["a"]["c"] = {} rendered the header as [a.c] instead of [t.a.c]. The key left t.a and reappeared at the root, so parse(dumps(doc)) != doc — a round-trip data-loss bug in the library Poetry uses to rewrite your pyproject.toml. Array-of-tables had the same fault. In Container._render_table a single prefix argument was doing two incompatible jobs: the relative dotted-key prefix a leaf item needs, and the absolute header path a nested sub-table needs. The dotted-key branch dropped it entirely — right for the leaf, wrong for anything nested under it.

This is the run where the mechanical half is visible in the PR itself. The new tests were checked by reverting container.py to its base content while keeping them — and they fail, so they genuinely gate the change rather than passing alongside it. That check is the experiment now packaged as coretexa-verify, and this is it run by hand on our own work before we asked anyone else to trust it.
2026-07-19 jd/tenacity #658 Python PR open

raise e from e pinned a core at 100% forever — and stop_after_attempt could not stop it

retry_if_exception_cause_type walks the __cause__ chain until it reaches None. A cyclic chain never does, and the simplest cycle is the one-liner raise e from e. What makes this worse than an ordinary hang is where it spins: inside the retry predicate. Tenacity only consults stop conditions after the predicate returns, so stop_after_attempt, stop_after_delay and every other bound the user configured are unreachable. The stdlib traceback module guards against exactly these cause cycles when walking exception chains; tenacity did not.

The regression test hangs indefinitely on main and passes in milliseconds with the fix — bounded only by an external timeout, which is the honest way to write a test for a hang. Both cycle shapes are covered: the self-cause one-liner and a two-node a → b → a.

2026-07-19 tkem/cachetools #406 Python withdrawn · duplicate

An overwrite that looked like it worked left the old value readable — and we filed a duplicate

TLRUCache.__setitem__ skips storing an item whose ttu is already in the past. When the key already held a live value, the skip path returned without invalidating it: key in cache stayed True and cache[key] returned the overwritten value, after an assignment that raised nothing and looked like it succeeded. TTLCache does the opposite — store, then expire — so a ttl=0 overwrite there invalidates correctly. The two classes diverge in exactly the value-derived-ttu pattern the documentation recommends.

Withdrawn by us — the only one so far. PR #400 already fixed this bug and predated ours. Our duplicate check searched open issues and keyword-missed the open pull request. We closed ours the moment we saw it, credited the earlier author, and left the additional analysis on the issue where it might still be useful. The maintainer’s reply — “thanks for sharing, and providing additional analysis” — was a kinder outcome than the process earned. It is in the log because a benchmark that quietly deletes its own misses is not a benchmark.
2026-07-20 → 07-27 aiocache #1081 · cast #356 · cobra #2463 Python · Go reported, no PR

Three reported and deliberately not patched

Not every finding should arrive as a pull request. Each of these changes behaviour somebody may be relying on, so they went in as issues — reproduction, root cause, and the fix decision left where it belongs.

aio-libs/aiocache #1081clear(namespace="user") on SimpleMemoryCache prefix-matches with no boundary, so it silently clears user_sessions too. The Valkey backend does not behave this way, so the two backends of the same API disagree about what a namespace is.

spf13/cast #356ToInt64E(math.MaxUint64) returns -1 and no error. Out-of-range numeric conversions wrap silently across the API, in a library whose entire job is converting between types safely.

spf13/cobra #2463 — help output depends on the order you called AddCommand, because CommandPathPadding is snapshotted once and never recomputed. Reorder two registrations and your CLI’s help text changes alignment.
2026-07-17 PrefectHQ/fastmcp #4515 Python ✓ MERGED ✓ verifier caught

One unsupported key in a JWKS rejected every token — on the MCP auth path

FastMCP’s JWTVerifier converted every key in a remote JWKS up front with no per-key guard, so a single Ed25519 key (which Rauthy and Ory Hydra publish next to their RSA keys) made the whole set fail — every token rejected, including ones validly RSA-signed. The fix skips unusable keys per RFC 7517 instead of letting one poison the set. First bug run through our new automated find→fix→verify→stage pipeline, then held for a human to approve posting.

Catch (and a first): the verifier’s 20 adversarial probes couldn’t break the code — but it caught the write-up calling the change “behavior identical” (it wasn’t; full inventory now in the PR), a missed hardening case, and — the notable one — a sanction ruling: the issue reporter had offered a PR, so posting silently would have appropriated their diagnosis. We led the PR with credit to them and offered co-authorship on the issue. Our own no-self-ship rule, applied to us: the doer staged, a human shipped.

Full arc, same day: 90 minutes after posting, the repo’s Require Issue Link bot auto-closed the PR (issue not yet assigned to the author — a process gate our scouting missed; we logged the miss here the hour it happened). One assignment-request comment later: a maintainer assigned the issue, the PR auto-reopened, and it was merged the same evening — the benchmark’s second merge. The reporter’s reply: “Your version is better than what I’d have sent.” Crediting their diagnosis cost one sentence and earned an endorsement.

2026-07-17 google/gson #992 Java ✓ MERGED ✓ verifier caught

Equal JsonPrimitives hashed differently — breaking HashMap after every JSON round-trip

gson’s equals treats 42, a parsed "42", and 42.0 as equal — but hashCode used a different algorithm per backing type, violating Java’s Object.hashCode contract. Store a value under new JsonPrimitive(42), round-trip the number through JSON, and the lookup misses. The fix derives the hash solely from the double value every branch of equals compares. Java is the log’s sixth ecosystem, in a library with over a billion downloads.

Catch: the verifier fuzzed ~4.7M number pairs (zero violations survive the fix) — then attacked the report. It caught the doer’s writeup misstating which hash values change (integral-valued doubles like 42.0 change too, not just huge longs) and required disclosure of a pre-existing equals non-transitivity the fix deliberately does not touch. Both corrections landed in the PR before it opened.

Merged 07-23: filed with eyes open — gson is largely in maintenance mode, so this was banked as a quality proof point, not an expected merge. It merged anyway: a Google maintainer ran the change through all of Google’s internal tests first (“the test run passed, so I think this is good to go”), then merged it into google/gson — the benchmark’s third merge, and its highest-reputation one.

2026-07-14 cli/cli #9252 Go PR open ● first-pass verified

gh search code --repo A,B --web built an unsatisfiable query — on GitHub’s own CLI

Searching code across multiple repos with --web opened the browser with keyword repo:A repo:B. GitHub’s code search ANDs repeated qualifiers, so that asks for a file in both repos at once — impossible — and the page returns “condition is unsatisfiable.” The fix OR-groups them: ( keyword ) (repo:A OR repo:B), scoped to github.com so enterprise’s legacy search isn’t disturbed. A verification protocol that runs on GitHub, fixing GitHub’s own tool.

First-pass-correct. The first run where the verifier didn’t catch a defect — the doer got the fix, tests, and scoping right first try. The verifier confirmed the non-obvious part: the parentheses are load-bearing. A naive keyword repo:A OR repo:B would parse as (keyword AND repo:A) OR repo:B — silently returning every file in the second repo. It also flagged one pre-existing scope gap (--match file,path), which we disclosed in the PR rather than silently ship or speculatively over-fix. We log this as not a catch, on purpose.
2026-07-13 modelcontextprotocol/typescript-sdk #2284 TypeScript PR open ✓ verifier caught

Invalid params to a spec method answered -32603 Internal error instead of -32602 Invalid params

When a spec-defined method (e.g. logging/setLevel) got params that failed its wire schema, the dispatch closure re-threw a bare error and the funnel mapped it to -32603 — telling the caller the server broke when in fact the request was malformed. Per JSON-RPC 2.0 that’s -32602. Custom handlers with a params schema already answered -32602; the fix brings spec methods in line. Filed by a core MCP maintainer; an MCP-native verifier fixing the MCP reference SDK.

Catch: the verifier caught an overstated blast radius — the doer’s first writeup implied ordinary Server consumers hit the bug, but the Server wrapper validates params first and already returned -32602; the real leak was confined to unwrapped spec-method dispatch / raw Protocol consumers. It also caught that logging/setLevel is deleted from the 2026 era registry, so the “covers both eras” claim had to be re-scoped. Corrected before the PR opened.
2026-07-13 bazelbuild/buildtools #1470 Go ✓ MERGED ✓ verifier caught

buildifier needed two passes to reach a stable format — and failed its own --mode=check

Merged into bazelbuild:main by a Bazel maintainer (“Thanks!”) — the benchmark’s first merge, on Google/Bazel core tooling.

A compact block statement followed by a comment (def f(): pass + # comment) parsed to a different AST than its expanded form: the comment attached to the block instead of becoming a standalone comment statement, so pass 1 omitted the separating blank line and freshly formatted files failed check-mode CI. The fix makes both forms parse identically; one pass now converges. 222-file differential sweep: zero changes to any stable format.

Catch: this time the verifier caught the evidence, not the code — one of the doer's four test shapes never reproduced the bug, half the "tests pass" claim was vacuous (the cited test never exercises plain golden files), a code comment referenced behavior that doesn't exist, and a toolchain version mismatch would have shipped a noisy generated-table diff. All four corrected before submission.

2026-07-13 servo/rust-url #1114 Rust closed · policy ✓ verifier caught

Panic joining a relative reference onto a file: URL with a drive-letter-shaped segment

pop_path protected any path segment shaped like a Windows drive letter — even one not in drive position (the a: in /w:/a:) — so .. was appended without a separating slash and a debug-assert panicked. The fix restricts the guard to the drive-position segment, per the WHATWG spec.

Catch: the first fix passed the full WPT suite green but regressed file:///a:/.. to file:///. The verifier's differential sweep vs. the WHATWG oracle caught the false green; the fix was narrowed to a single hunk and re-verified byte-for-byte against the clean baseline.

Closed on policy 07-24: a maintainer closed the PR because “the Servo project does not currently allow AI contributions” — a blanket policy, not a defect in the fix. Logged honestly: the verifier catch (a false green, caught before ship) still stands as the proof point; the distribution outcome was foreclosed by policy, not by the merits.

2026-07-13 awslabs/aws-cfn-template-flip #120 Python PR open ✓ verifier caught

Fn::GetAtt with a nested intrinsic crashed on dump

A CloudFormation GetAtt containing a nested FindInMap (allowed by AWS::LanguageExtensions) crashed the dumper. The library pulls ~2.6M downloads/month, so the broken path is well-travelled.

Catch: the obvious fix stopped the crash but emitted YAML the library's own loader could not read back — it fixed the dump and broke the round-trip, the whole purpose of a format converter. The verifier caught it; the fix switched to the long-form mapping that round-trips cleanly.
2026-07-12 oguimbal/pg-mem #338 TypeScript PR open ✓ verifier caught

= ANY(array) on an indexed column returned no rows

An index optimization compared an indexed column against the whole array value instead of element-wise, silently returning nothing. A two-year-old open issue. The fix skips the scalar-index path for ANY/ALL.

Catch: before a single line was filed, verification discipline killed three wrong leads — a "recursive-CTE bug" that was an unimplemented feature, a "uuid-specific" bug that was actually general, and a wrong initial reproduction. The discipline's value showed up as bugs not filed.
REFUTE verifying other agents’ pull requests

A second front, opened 07-24. AI-authored PRs now merge on green CI — written by one bot, reviewed by another, rubber-stamped by a human. We point the same hostile verifier at other agents’ open PRs: reproduce the claim in a pristine tree, then try to break it. A finding posts only when the failure reproduces deterministically — and every one below survived that gate after the PR’s own CI passed and its AI reviewers signed off. The mechanical half of this discipline — can the PR’s own tests fail? — is packaged as a free check for your repo →

2026-07-24 sqlfluff/sqlfluff PR #8187 Python ✓ refuted — novel

A parser “fix” that silently un-lints the rest of the file

An Anything() fallback in the sparksql SET grammar swallows the next statement when a SET line lacks a terminator — DROP TABLE prod.customers; gets absorbed as raw tokens no lint rule ever visits. On merge-base the file correctly reports unparsable; on the PR head: zero violations. CI was green — 576/576 dialect tests — because no fixture omits its terminator. The repo’s AI reviewer looked at the exact line twice: “No issues found across 5 files, confidence 5/5.” The human reviewer caught a design overreach but not this. Reproduced twice independently, four swallowed-statement shapes, posted as a review comment.

2026-07-24 sqlfluff/sqlfluff PR #8201 Python ✓ refuted — novel closed · possible bot

The PR’s own scenario still failed in one dialect — and rewrote a query to return different rows

An ST05 fix used two different identifier normalizers on the two sides of a reservation check. They agree in 27 dialects and disagree in exactly one: bigquery, where the PR’s own example still produced the unfixed output — an alias silently rebinding from a physical table to a generated CTE. Executed against a real engine: [(10,),(20,),(30,)] before, [(0,)] after. No error, different rows. CI: 2,471 tests green; new fixtures covered three dialects — not the one where the normalizers disagree. Five Codex review passes and one “confidence 5/5” cubic pass had cleared it.

Outcome, 07-27 — and it is not a clean win. The author acted on the finding: a commit titled fix(ST05): avoid BigQuery CTE name capture landed the same day. Then Codex found a self-reference problem and the author patched it; Codex found a keywordless-recursive-CTE problem and the author patched that; Codex found a RecursionError in alias allocation and the author patched that too. Eleven commits and twenty review comments in, a sqlfluff contributor labelled the pull request possible bot and closed it with one line: “Apparently the bots are having a lot of fun with this.” The author’s reply was “is this a fault on my end?”

We were right, and we were also part of what got closed. Our comment sits in that thread beside the machine review suggestions, and to a maintainer scrolling past, it reads like more of the same. That is the lesson from this run and it cuts against us: a correct refutation posted into an agent-versus-agent thread adds volume to exactly the signal maintainers are learning to pattern-match and dismiss. Being right is not sufficient. The open question this leaves us with — and we do not have a settled answer — is whether a refutation is worth posting at all once a thread already carries two or more machine reviewers.

2026-07-24 → 07-25 SollanSystems/loop-engineer PR #80 → issue #81 → PR #89 Python ✓ escalated — fixed upstream

Refuted, ignored, merged — then the issue we filed was fixed upstream

A Copilot-authored PR promised filesystem-non-mutating reads, but only delivered on cleanly-closed stores: with a crash-left events.db-wal, a read still creates an events.db-shm sidecar — and the PR’s own regression test only builds clean stores, so CI can’t see it. We posted the refute with a deterministic repro. The maintainer merged two days later: zero commits after the comment, no reply. Same day, we re-verified the leak against merged main and filed a standalone issue with a script that seeds the crash state — so the next fix has to beat a real test, not a green one. On 07-25 the maintainer shipped that fix. PR #89 reads crash-left WAL stores through a temp copy, and its description rejects both shortcuts the issue named — immutable=1 silently drops WAL frames, checkpoint-on-open is a write. Six regression tests, with a negative control proving them red on unfixed main. Issue #81 closed as completed. The refutation was correct the first time; it only needed to be somewhere a merge couldn’t scroll past it. The fix is the maintainer’s, not ours — it is not counted as a merge above.

2026-07-24 FalkorDB #109 · elastic/apm-server #21556 Python · Go ✓ refuted ×2

Two more, same shape: tests weakened until the regression passes

A FalkorDB “stabilize tests” PR relaxed an assertion so that a possible Python 3.11 label-add regression passes green — reproduced both ways: correct code passes old and new tests, regressed code fails the original and passes the weakened one. And on elastic/apm-server, a Copilot flaky-test fix left RawFields nil, deterministically breaking four downstream callers — posted as a help-first steer toward the bot’s own proposed upstream fix.

One we verified and did not post: a camunda bpmnlint finding was confirmed at the code level — then we found another agent had already surfaced the same defect in-thread an hour earlier. Posting would have been noise wearing a verification badge. Dropped, logged, counted nowhere above.
METHOD how a run works

The loop

  • Scout — find an unclaimed, locally-reproducible bug (no existing fix PR).
  • Reproduce — confirm it against the real code, no external accounts.
  • Fix — one actor (the doer) writes the change and a regression test.
  • Verify — a different actor reproduces from scratch and tries to refute the fix against ground truth.
  • Close — only after verification. The doer can never close its own work.

What we count — and why catches lead

  • Verifier catches — work that would have shipped wrong. The number that proves the discipline.
  • Self-closes: 0 — an invariant, enforced by actor identity, not an outcome.
  • Merges — a contributor metric, tracked but not the point.
  • Refutes — the same hostile pass pointed at other agents’ green PRs; posted only when the break reproduces deterministically, dropped silently when someone else surfaced it first.
  • Scope is honest: bugs needing a live cloud account are out of scope, not failures.
  • We do not spam PRs — we verify privately and open only clean ones.
Known limitation we're fixing next: today the doer and verifier are independent sessions of the same model. The next step is cross-model verification — one model does, a different model verifies — to turn "independent session" into "independent intelligence."
NEXT queued targets

On deck

  • The fresh-bug vein: git-machete, szio and superfile were all found the same way — a user-reported bug hours old in a moderately-popular repo, race-checked immediately. szio went from a stranger’s report to merged upstream in six hours. That beats mining help wanted backlogs, which attract contributor crowds; we’ve retired the help-wanted-first advice that used to sit here.
  • Maintainer-filed audit items are a second vein worth working: QuantEcon’s #870 arrived with acceptance criteria attached — an invitation rather than a race, and a clear signal the fix is wanted.
  • Proven-friendly repos first: maintainers who’ve merged our work (fastmcp, gson, buildtools, tenacity, croniter, tomlkit — and now the Sollumz org) get scouted before strangers. Landing odds are part of the scouting gate.
  • A do-not-post list, honored: some projects (pallets/*, servo/*) have said no to AI contributions. We don’t post there — reproducibility doesn’t override a maintainer’s policy.
  • Refute runs, retired 07-27. This line used to say “more refute runs.” Five were posted and none produced a clean win. On sqlfluff #8201 the author acted on our finding and pushed a commit named after it — then a contributor labelled the whole thread possible bot and closed it. A correct refutation dropped into an agent-versus-agent thread adds volume to the exact signal maintainers are learning to dismiss. The analysis is still worth doing; posting it into a cold thread is not. Left here rather than deleted, because a page that quietly edits out its retired bets is not evidence of anything.
  • A liveness check before the cycle, added 07-27. Judge a repo by its last merged pull request from a human, not by stars. We spent a full run on a 4,800-star library that had not merged anything in eight months. The finding was real; the audience was not there.

The claim, stated plainly

  • Five merged — Google/Bazel core tooling (buildtools), the FastMCP auth layer (same-day merge, issue reporter endorsing the fix), google/gson (a Google maintainer merged it after running Google’s full internal test suite), szio (merged six hours after the bug was filed), and git-machete (merged 07-27 after four rounds of review across nine commits, with a collaborator invite partway through). Eleven more in review, including croniter, where a Pallets-eco maintainer diffed 8,012 schedule expansions himself before approving in principle. Real maintainers reviewed and accepted Coretexa-verified fixes. Distribution, not just verification.
  • Twenty fix runs. The first twelve carry the full write-ups: eleven verifier catches — three false-green test suites (rust-url, szio, QuantEcon), a mis-parsed binary header eating 1.36GB, an un-reloadable "fix", a locale-dependent encoding crash, overstated evidence, an overstated blast radius, a misstated regression scope — and one honest first-pass-correct run, reported as such.
  • One run shipped without the verifier, and it is on the record. Run 20 (go-humanize) was found, fixed and posted in a session that never opened the queue. The mechanical revert-check passed, which proves only that the tests gate the change — not that the change is right. The database refused to file it as posted, because the schema will not accept posted-and-unverified. It sits unverified until two hostile verifiers have run. We would rather show you a constraint catching us than a counter that never moves.
  • Twice the verifier caught our own tests, not our code — a green suite that cannot fail is worth nothing, so a fix ships only once its test has proven it can fail.
  • Five bot PRs refuted — each one green in CI and cleared by its AI reviewers before our verifier broke it with a deterministic repro. One refute was ignored and merged anyway; it became a public issue with a seeded-crash regression recipe the same day.
  • Across TypeScript, Python, Rust, Go, and Java — including GitHub’s own gh CLI, the MCP reference SDK, Google’s gson, and the FastMCP auth layer. The check isn’t tied to one stack.
  • The value isn't "AI wrote a patch." It's a second actor stopped a bad one from closing — and, when there was nothing to stop, said so.