“A Checker That Cannot Edit The Tests” Is Right. Permission Rules Are Not The Lock.
A fortnight ago we took apart a loop-engineering card whose off-switch was a 95 out of 100 awarded by a model. The problem was structural: the evaluator could not open the file it was scoring, so the loop stopped when a cheap model said it was happy.
This card is the fix, and it is worth saying so before anything else. Its verifier is not an opinion:
“A test suite that exits 0 or 1.”
“Score worse?
git reset, move on.”“If it can edit the test, it will quietly make the test easier.”
That third line is the whole insight, and it is correct. An exit code cannot be flattered. The only way to game it is to change the test — which is exactly what the card then tells you to prevent.
So this piece does two things. It checks the four numbers the card prints and declines to verify. And it looks at the one mechanism the card recommends for stopping the checker touching the tests, because the documentation has a sentence about that.
Source card: “How to do Loop Engineering” — BANI Academy, 22 October 2026. Split the roles, three files only, run the ratchet, the payoff.
View the original post on Facebook
The four numbers, checked
The card’s footer says plainly: “Figures come from the source image and are not verified here.” Fair enough — so we verified them against the primary source, which in this case is a public repository and its author’s own posts.
| The card’s figure | What the source says | Verdict |
|---|---|---|
| 700 experiments run by the loop | About 700 experiments, run over roughly two days on a depth-12 model by a tool called autoresearch | Confirmed |
| 20 wins kept, all others reset | About 20 changes that improved validation loss — and the author adds that they were additive and transferred to larger depth-24 models | Confirmed, with a detail the card leaves out that makes the result stronger |
| 11% faster, 1.80h down from 2.02h | The nanochat leaderboard lists the 1.80-hour entry as “autoresearch round 1” | Confirmed |
| A missing QK-Norm scaler | The author describes a parameterless QK-norm that lacked an attached scaler multiplier, leaving attention too diffuse | Confirmed |
| “Each run takes about 5 minutes” | Not stated directly — but 700 runs across two days is about 4.1 minutes each | Consistent |
One thing the card omits is the author’s own framing. He described the output as not novel, ground-breaking research — while adding that the adjustments are real. That is a useful calibration to carry into step 4: the loop found genuine, transferable improvements to a known codebase. It did not invent anything.
Now the lock
Step 1’s instruction is “Add a reviewer with /agents that can never edit tests”, with the diagram label “/agents reviewer can’t edit tests”. The principle is right. The question is what enforces it.
Start with what the checker must be able to do. Its job is to run prepare.py and read the exit code. That requires Bash. You cannot take shell access away from the thing whose entire purpose is to execute a test suite.
And here is what the permissions documentation says about Bash rules, quoted in full:
“A Bash rule matches the command text Claude writes… It doesn’t match the same program invoked in a different form, so a deny or ask rule covers the invocation Claude usually produces and isn’t a security boundary around the program.”
Its own table makes the point concrete: Bash(rm *) stops rm -rf build/ and does not stop /bin/rm -rf build/ or bash -c 'rm -rf build/'.
Diagram: three ways the lock does not hold
| How you would naturally write it | What the documentation says | What to do instead |
|---|---|---|
Deny Write(prepare.py) or Write(tests/**) |
“Write(path), NotebookEdit(path), and Glob(path) rules never match. Claude Code checks file permissions against Edit(path) and Read(path) rules only… Claude Code accepts the rule but never matches a read or edit against it” |
Write Edit(prepare.py). The rule you would reach for first is accepted and silently inert |
| Deny the shell commands that could overwrite it | A Bash rule “isn’t a security boundary around the program”; another invocation form is not matched | Stop trying to enumerate commands. You cannot list every way to write a file |
| Trust the path rule to cover the test run | Path rules do not apply to “arbitrary subprocesses that read or write files indirectly, like a Python or Node script that opens files itself” | This is the checker’s actual job. Running prepare.py is a Python subprocess that can open files |
| — | “For OS-level enforcement that blocks all processes from accessing a path, enable the sandbox.” And: “For filesystem and network enforcement that doesn’t depend on the command text, use sandboxing” | The sandbox is the lock. Or keep the tests read-only on disk, outside the agent’s reach |
The second row deserves emphasis because it is the quiet one. A deny rule for Write(prepare.py) does not error. It does not warn. It sits in your settings file looking exactly like protection, and it matches nothing. If you build this loop and write that rule, you will believe the checker is constrained when it is not.
None of this means the card is wrong. It means the sentence “a reviewer that can never edit tests” describes a goal, and tool permissions are the wrong layer to achieve it in. The right layer is the operating system, which the documentation names in one line.
The cheapest version of the same guarantee
If a sandbox is more setup than you want for a Saturday experiment, the card’s own three-file structure already suggests two simpler locks:
- Keep
prepare.pyout of the working directory. Claude Code’s file tools are confined to the directories you give it. A test file that lives somewhere the agent was never pointed at is not reachable by the ordinary path. - Commit the test and check it after every cycle. The ratchet already runs
git reseton a worse score. Add one line:git diff --exit-code -- prepare.py. If the test changed, the run is void. This costs nothing and catches the exact failure the card is worried about.
The second is in the spirit of the card. Its whole argument is that you verify with something that exits 0 or 1 rather than something that forms an opinion — so verify the verifier the same way.
What the rest of the card gets right
| Element | Why it works |
|---|---|
| A verifier that exits 0 or 1 | The single best idea on any loop card we have read. It replaces a judgement with a fact, which is what the earlier card we checked was missing |
Only train.py ever changes |
A one-file blast radius. Everything else is either yours to write or read-only, so git reset is always clean and always total |
The ratchet: worse score, git reset |
Version control as the undo button. No cleanup logic to get wrong, and no partial state to debug |
Log the idea, result and lesson to results.tsv |
The part most people skip. Without it, 700 runs leave you with one number instead of a record of what was tried |
| “Maker never grades. Checker never writes.” | Separation of duties, which is an old accounting control and the correct instinct here. The implementation is the only thing in question |
| The footer | “Figures come from the source image and are not verified here. A loop like this still needs a human to review the results” — honest about its own sourcing, which is rarer than it should be |
One small inconsistency
Step 3’s diagram says “100x repeat” and step 4 is titled “Let it check 100 times” — while the panel beside it reports 700 experiments. Both are from the same run. A hundred is almost certainly standing in for “a lot”, and the real figure is seven times that. Worth knowing if you are budgeting compute from the card rather than from the source.
Four honest caveats
- We did not build the loop. The verification above is of the card’s figures against their source, and of its permissions claim against the permissions documentation, in October 2026.
- A subagent’s tool list is genuine enforcement. Removing
Editfrom a checker’s tools really does remove it. Our point is narrower: the checker needs Bash to run the suite, and Bash is where the documented gaps are. - “Not a security boundary” is about a cooperative agent, not an attacker. Nothing here suggests a model is trying to cheat. The failure the card names — quietly making the test easier — is a gradient-following accident, and accidents still need a lock.
- This is the best loop card we have taken apart. A deterministic verifier, a one-file blast radius, a git-based ratchet and a results log are four correct choices in a row, and its numbers survived checking. The objection is one layer, not one design.
The one-line version
Replacing a model’s 95-out-of-100 with a test suite that exits 0 or 1 is exactly the right move, and the card makes it — but “a reviewer that can never edit tests” is a goal, not a setting: a Write(path) deny rule silently never matches, a Bash rule “isn’t a security boundary”, and the checker needs a shell to do its job.
Use Edit(path), not Write(path). Keep the test outside the working directory, or turn the sandbox on. And add git diff --exit-code -- prepare.py to the cycle, because the card’s own argument is that you check with exit codes.
Source card: “How to do Loop Engineering” — BANI Academy, 22 October 2026.
View the original post on Facebook · Related: what /agents actually gives you, and who holds the plan
