How a verdict is made.
SCAR does not read a patch and form an opinion. It compiles the fix, puts it where the attack lands, replays a reproduction of the real exploit on chain state frozen at the incident, runs the checks the fix was never shown, and hashes the result.
Freeze the chain one block before the exploit.
Every JSON-RPC response the exploit and the checks need is recorded once, then served offline by a replay proxy that fails closed: a request that is not in the recording gets an error, never a live answer. A case is built by re-recording until an offline replay runs the exploit and every check without missing state.
Foundry’s own fork cache is switched off, so the recording is the only source of chain data, and the containment backend runs the grader in a container with no network at all.
MCAI, for exampleEthereum mainnet at block 21,720,380, served from 134 recorded responses.
Remove half of them and the same reference repair grades inconclusive instead of solved: see the receipt.
Put the fix where the attack lands.
The patched source is compiled and its bytecode placed on the exploited contract’s live address, keeping the live storage. The fix, not a mock, decides whether the exploit still works.
A patch may touch only the contract source. The exploit test, the hidden suites and the build configuration are hash-locked and re-verified before every run; a patch that reaches outside is refused before anything executes.
ScopeThe placed bytecode is a recompilation of the verified source, not the deployed bytecode, so a grade is a property of the task’s contract. Proxies,
immutablevariables and unverified sources are outside the current build pipeline.Replay the historical exploit, in phases.
The proof of concept comes from DeFiHackLabs, rewired to run against the placed fix, and is split into preconditions, the attack, and a profit check against a threshold set per case (11 ETH for MCAI, which lost 12.03 WETH). The exploit counts as blocked only if it fails in the attack or the profit step, for a reason the task declares.
A failure anywhere else, such as a sabotaged view function or an undeclared revert, is not a block. That is how a fix that breaks
allowance()to make the exploit “fail” gets caught.Try itSabotaged allowance view: the exploit test fails in its precondition checks, so it is not counted as blocked, and all four security obligations show the bypass is intact.
Check what the fix was never shown.
Security obligations state the vulnerability class rather than the one attack: each fails on the original contract by construction, and a failure confined to them marks an incomplete repair.
Legitimate-use obligations are hand-written tests of what ordinary holders, traders and operators do, plus generated checks that the original functions are still dispatched, with two guards that stop those checks from passing vacuously. None of this replays recorded protocol traffic: legitimate use is specified, then tested.
The set of tests that runs must match the task’s manifest exactly. They are hidden from whoever writes the fix and published in the repository, so they work as a specification, not a secret.
Why both15 of the 48 test fixes block the exploit and still fail: some leave the class open, some break what users rely on. The exploit alone cannot tell them from a repair.
Flag what should never pass.
Canaries force a score of zero and are recorded on the receipt:
- residual vulnerability: legitimate use works, but a security obligation fails
- bricked: the exploit test fails, and a legitimate-use or interface check fails too
- empty patch: nothing changed
- out-of-scope edit, PoC tampered, harness tampered: the patch reached for the test instead of the contract
ScoringThe grade is read from Foundry’s structured test report and the process exit code, never from its human-readable log (a text fallback exists for broken reports, and it can never produce solved).
“Could not tell” is an answer.
A verdict is solved, not solved or inconclusive. Inconclusive is reported on its own and never folded into either side: for training it is a missing observation, not a negative label.
The exploit is blocked for a declared reason, every hidden security and legitimate-use obligation passes, the ABI is preserved, the patch stays in scope and no canary fires.
The run produced valid evidence and the fix does not qualify: the exploit lands, the repair is incomplete, or it breaks legitimate use. The receipt says which.
The run produced no valid evidence: a failed build, missing recorded state, a crashed runner, an unexpected test set. Typed reasons include compile_failed and unrecorded_rpc.
Two hashes. Anyone can recompute them.
A receipt is the SHA-256 of a canonical form of the grade. It proves which grade you are looking at, not that the grade is right, and it identifies a grade, not a patch. To check the grade, re-run it.
core
The decision and its evidence: outcome, typed reason, the exploit result, the ABI checks, canaries, changed files, and every executed test id with its status.
strict
Everything in core, plus the revert or assertion message of every failing test. For the same patch, a core match with a strict mismatch points to a toolchain difference, not a different grade.
Left out on purpose
Timings, gas, ports, temp paths, raw logs, exit codes and the backend name: properties of the run, not of the grade.
Checked across backends and machines
For every case, the reference and original grades hash identically on the local backend and in the no-network container (16 hashes, 8 grades). Every push also re-grades all controls on GitHub’s Linux runner and fails if a committed hash moves.
One caveat, stated on the receipt. A complete run reproduces byte for byte. An inconclusive run records whatever partial evidence it got, and that can differ between runs. On 23 Sep 2026 every grade except the Docker one was re-run on two machines: a fresh clone on the Mac that recorded them, and GitHub’s Linux CI runner, which now re-grades every control on each push. Every hash reproduced on both, except the missing-state run’s, which kept its outcome and reason.
What a verdict does not say.
A verdict is scoped to the obligations written for its incident. The limits are part of the method.
- Tests, not proofs.Obligations are finite tests of the declared class. They are not a formal proof of security, and they have not been shown robust to every adaptive fix.
- Declared reasons are literal.A correct repair that makes the exploit fail with a message the task does not declare grades inconclusive, never not solved, and should be triaged by hand.
- Written by us.Obligations and controls for these four incidents were written by SCAR. Independent qualification of new incident families is the next step.
- One toolchain.Grades are produced with Foundry 1.7.1 and pinned compilers. A Foundry upgrade can change failure messages and would move the strict hash: a deliberate, reviewable event.
- Thresholds, not zero.The replayed exploit counts as landing only above a profit threshold per case (for NGP, 100,000 USDT against a loss of about 2M). A fix that only shrinks the attack is left to the security obligations, as the drain-capping near-miss shows.
- Recorded state is literal.A correct repair that reads chain state the recording never captured grades inconclusive, never solved. MCAI’s alternative repair grades solved only because it rejects a zero sender before reading any allowance.
When the code changes, the verdict has to follow.
Revision review re-runs the checks on each new revision, refuses to carry an old result forward and blocks approval when a check disappears.