Don’t trust the fix. Replay the attack.
AI agents now write smart-contract security fixes in minutes. SCAR tells protocol teams, audit firms and bounty platforms which ones hold: we fork the chain one block before a real exploit, replay the attack against the fix, test that legitimate use still works, and return a verdict with receipts.
Fork one block before the exploit.
Ethereum mainnet frozen at block 21,720,380 and served from 134 recorded RPC responses. The network is off.
Run the attack twice.
Once as it happened, once against the fix — with four hidden security obligations and twelve legitimate-use obligations.
Ship the fix that holds.
Solved, not solved or inconclusive — with a canonical hash anyone can reproduce.
- Fix Verifier, Apache-2.0 — 4 historical incidents on Ethereum and BNB Chain, 48 graded control patches
- Public datasets on Hugging Face — 231,269 contracts, 7,552 auditor-finding pairs
- An RL environment on the Prime Intellect Environments Hub
- Hosted verification for the first pilot teams — requests open
- 24 independently qualified Ethereum incident families
- A public leaderboard with re-graded entries
- Verdicts that follow the code through each release — previewed on public OpenZeppelin and Safe changes
- 48 incident families, and Solana
- Verdicts as attestations and underwriting inputs
Exploits hit a record. Fixes are still graded on trust.
Agents now write and patch the code that holds money. Review built for humans takes weeks, the vendors score themselves, and “fixed” usually means a test suite still passes — not that the attack stopped working.
hacks in the first half of 2026 — the most TRM Labs has recorded in any six-month period.
of them exploited smart-contract code, the most common kind of incident.
the incident count of the same half-year in 2025, when TRM recorded 83.
Source: TRM Labs, H1 2026 crypto hacks review. Figures are incident counts, 1 July 2026.
“Fixed” should mean the attack fails.
SCAR does not read the patch and form an opinion. It runs the real exploit against the patched contract on chain state frozen at the incident, then checks what the fix was never shown.
Fixed, as it is usually graded
One check, written by whoever wrote the fix.
- ✓The test suite passes1 check
A fix that stops the demo exploit and leaves every other holder exposed passes this check.
Fixed, as SCAR grades it
The real exploit, plus sixteen hidden checks, on the real chain state.
- ✓Historical exploit replayed at block 21,720,380blocked
- ✓Hidden security obligations4 / 4
- ✓Legitimate-use obligations12 / 12
- #Canonical grade hashf34d89ba…963e
Or not solved. Or inconclusive — a missing piece of state is never counted as a pass.
Two of these fixes stop the exploit and still are not repairs.
Five candidate fixes for the MCAI incident, re-run on 20 September 2026: the four complete grades matched their archived hashes, and the missing-state run matched its outcome. An exploit-only check accepts both of those two. SCAR rejects them and names the checks they fail. The run below starts on the incomplete repair, one of the two. Pick any fix to replay its recorded grade with the verifier’s real test names and failure messages.
Select a patch to replay its recorded grade.
strict —
$ python -m evmpatch_env.sandbox tasks/mcai_2025_01 --patch worked_example/mcai_2025_01/controls/pair_only_allowance_fix/Token.sol --backend local --sha256
The exploit is blocked, every hidden security obligation passes, and legitimate use still works.
The exploit lands, the repair is incomplete, or it breaks legitimate use. The receipt says which.
Missing state or a failed build. Reported on its own, never credited as a pass.
| Patch | Exploit-only check | SCAR verdict |
|---|
Exploit-only check: what a grader holding only the exploit test would say. On the missing-state run the exploit test still fails with a declared reason, so it would credit a run with half the evidence missing.
Already in use.
The open core ships first: it is how practitioners find SCAR, and how anyone can check that the verdicts are what we say they are.
“The model he trained helped me retrieve relevant vulnerability patterns more effectively and push the recall rate even higher during the competition.”
contracts in the public corpus, released on Hugging Face in April 2026.
auditor-finding pairs, with 838 evaluation pairs, in the same release.
Listed on
- Prime Intellect Environments Hubevmpatch-env
- Hugging Face4 datasets · 1,390 downloads
Grant
- Ethereum Security QF round2026
Figures as of September 2026.
One trusted check. Three products run on it.
Incident cases go in; verdicts with receipts come out, for the teams shipping fixes, the vendors being scored and the agents learning to repair.
Incident cases
Real exploits on frozen chain state. A case counts only once its exploit is reproduced offline, its hidden security and legitimate-use obligations are written, and its control patches grade as intended.
Live · 4 incidents · 48 controlsFix Verifier
Replays the attack against a fix, checks the hidden security and legitimate-use obligations, and returns solved, not solved or inconclusive — with a receipt anyone can replay.
Live · open sourceHosted verification
A verdict on the fix you are about to ship or sign off, re-checked when the code changes. Per verification.
Next · pilots openRe-graded board
Every tool scored on the same cases. A subscription buys re-grading, never rank.
NextTraining environment
A reward that is executed, not guessed. Open source.
LiveEach qualified incident is used three ways: to verify a fix, to score tools on the planned board, and to train repair agents. Every verdict is a hashed receipt that replays offline. Bound to the exact patch and signed, a receipt can become an attestation a counterparty checks before it trusts a fix, as agent standards add validation registries such as ERC-8004.
Smart contracts first
Their exploits are public, replayable and priced in dollars, so a verdict can be checked against the real attack.
The method, not the chain
Replay the attack, check what the fix was never shown, hash the result. It applies wherever an attack can be reproduced.
Whoever signs off a fix
Protocol teams before a fix holds money, audit firms and bounty platforms for independent review, AI audit vendors for re-grading. Public grades stay free, and no payment ever touches one.
Models can write fixes. Graders cannot tell which ones hold.
- Models write fixes. DeepSeek V4 Pro repaired a real exploit in 10 of 16 graded attempts.
- Graders over-credit them. A public patch benchmark gave an open-weight model 11 credits; 5 were real repairs. Another benchmark’s grader accepted 20 patches; 13 could not be shown to preserve legitimate use.
- SCAR tells them apart. We attack our own verifier first: 15 of our 48 control patches stop the exploit and still fail.
DeepSeek V4 Pro
16 attempts across the four incidentsEvery incident solved at least once. Hard enough to matter, tractable enough to train against.
Qwen3.8-27B, open weights
A public patch benchmark gave it 11 credits. We read every one.A credit is not a repair until somebody checks.
Of the 48 graded controls, 26 block the exploit. Only 11 grade as complete repairs. SCAR marks the other 15 not solved: 8 incomplete, 7 that break the protocol.
Four exploit classes today: broken access control (MCAI, Ethereum), reward accounting (GoldReserve, BNB Chain), unbounded batch payout (Bitallx, BNB Chain) and AMM reserve manipulation (NGP, BNB Chain).
of files in a public AI-audit benchmark have a near-identical twin (token similarity ≥ 0.8) in public contract corpora — 122 exact. A lower bound.
patches an execution-based benchmark’s own grader accepted could not be shown to preserve legitimate use. 5 did; 2 broke it. It counts an out-of-gas replay as “behaviour preserved”.
queries in SCAR’s held-out evaluation set overlap those corpora, under the same test.
Bring us a fix you are about to ship.
We will replay the incident against it, check that legitimate use still works, and send you the receipt — including the one that says we could not tell.
Request a pilot verification
Send the incident, the patched source and the calls that must keep working. Get back a verdict, the failing checks by name, and a receipt you can re-run.
[email protected] Agent builders · researchersTrain against the verdict
An execution-verified environment with a reward that is not a guess.
evmpatch-env on GitHub Investors · partnersTalk to us
Where the verification layer goes next.
Start a conversation