Checking the cage

The result in one paragraph

An AI evaluation sandbox has a catalogued set of ways out. We fixed that set at 48 routes, drawn from the published literature rather than from memory. Nullmoor's 22 probes fully watch 36 of them; 4 more are watched in part and 8 are not watched at all. Every probe is matched to a broken environment it has to detect, and a blinded copy of every probe is caught by the suite itself. That is the whole claim. It is deliberately narrower than the claims usually made in this area, and the sections below say exactly where its edges are.

Why the sandbox is the weak point

AI systems are increasingly permitted to act — to run code, open sockets, read files, and drive tools. The safety and capability claims made about them are produced inside sandboxes and reported as though the sandbox were a given.

It is not. A sandbox specified correctly and deployed incorrectly returns results that are indistinguishable from correct ones. A subject that quietly reached the host during an evaluation invalidates every number in the report, and nothing in an ordinary evaluation pipeline would notice. The model is checked closely. The cage around it is usually assumed.

The most directly relevant prior work, SandboxEscapeBench, measures whether frontier models can break out of container sandboxes, and its finding is blunt: when a vulnerability is present, models find and exploit it. That work asks can this model get out? Nullmoor asks the complementary question, is this cage shut? — and the second is cheaper to run continuously, because it does not need a frontier model or a capture-the-flag setup to notice that a control socket has been mounted in.

The catalogue: 48 doors

Guessing at what a cage needs repeats the mistake that lets things out: the routes that matter are the ones nobody thought of. So the starting point is an enumeration assembled from the literature. It uses the four-way division from SandboxEscapeBench, which fits the field.

Enumerated route by route, the catalogue holds 48 entries. Two of them are the same capability seen from two angles, so 47 are distinct; the numbering is kept so earlier references stay valid. The wider security field frames the whole family as MITRE ATT&CK T1611, Escape to Host, and NIST SP 800-190 names container escape the critical case, because the kernel is shared.

The method: every probe must be able to fail

A check that cannot fail is not a check. It is the single rule the whole method rests on, and it has three consequences that shape everything Nullmoor does.

A probe reports a seal only after trying

Each probe attempts the thing the door would allow, and reports SEALED only when the attempt is refused. It does not read a configuration file and infer that the door is shut. A configuration file is a statement of intent; the run-time behaviour is the fact, and only the fact is reported.

Every probe is matched to a broken environment

For each probe there is a computed environment in which the door it watches is genuinely open, and the probe is required to detect it. A probe that has never caught anything has not been shown to detect anything. The set of these matched pairs is the corpus.

The checker is checked

A blinded copy of each probe — one rewritten to report SEALED unconditionally, without looking — is run through the suite, and the suite is required to catch every one. This is the guard against the most dangerous failure in the field: a check that quietly always passes. Separately, a stand-in subject attempts to leave a sealed environment by routes the probes do not use, as an independent test of the checker.

What we found by attacking our own checker

The method is only worth the audits it survives. The reference cage was taken and broken one way at a time — 17 single misconfigurations of an otherwise hardened container. The suite reported 12 correctly. On 5 it reported sealed while the hole was real.

The common cause was the same each time: a probe that checked a fixed list of known locations. A misconfiguration does not have to use a standard path, and a list is only ever as good as the list. The host filesystem mounted read-only rather than read-write; a control socket mounted at an unusual path; a credential mounted outside the usual directory; the host kernel-log device passed in; a raw block device passed in. Each was reported sealed because each sat one step off the path the probe knew to look at.

All five are now handled, each held to the method's own bar — a matched broken environment and a blinded control — not a spot check:

• A new probe walks the container's device directory and reports any host block device node or a readable kernel-log device, at any path, rather than a known few.
• The mount probe now reports a host filesystem mounted read-only as well as read-write; read access alone exposes the host's files.
• The runtime-socket probe now reports any host socket mounted into the cage wherever it sits, in addition to the known runtime paths.

One residual is left open on purpose and named rather than closed: a credential baked into the image itself at an unusual path. A scan wide enough to find it would report a warning on nearly every container, which is the habit this method exists to avoid, so it is declared open instead of covered by a check that would rarely mean anything.

Receipts. Against the reference cage (WSL2, Docker 29.1.3): the corpus covers all 22 probes with a matched pair, 0 failures; the blinded self-test catches 24 of 24; the end-to-end acceptance matrix passes 19 and fails 0; 44 unit tests pass, and a test fails the build if the published coverage count and the catalogue table disagree.

Coverage, as a fraction

Nullmoor fully watches 36 of the 48 catalogued routes. It is strong on privilege allocation, which is one cheap local file read and decides which of the other escapes are even possible, and on the misconfiguration routes above. The remainder is not rounded away.

Two of the unwatched routes are left off the build list deliberately, not for lack of time. Matching the kernel version against a list of vulnerable runtimes would be a different kind of claim from every other probe — not I tried this and it failed but I read a version number and consulted a list I shipped. That list is stale the week it is written, and a stale list that reports a seal is precisely the failure this project exists to prevent; if it is ever built it must report as inventory, never as a seal. Reaching the grading harness is the highest-value target in the room and the least generic: it depends entirely on how a given lab wired its evaluation, and it needs a real harness in front of it rather than speculation.

What this does not establish

The stand-in subject found three routes out because it was told to try three things. It is a script, not an adversary; it tries what it was told to try. This catalogue is the same shape — a list assembled from what other people have already written down. Neither finds the door nobody has published yet.

That is the honest position, and SandboxEscapeBench is the warning that makes it necessary: when a hole exists, a capable model finds it. A list of known doors is necessary and it is not sufficient. A sandbox calibrated to what can be enumerated today is a floor, not a ceiling.

And the scope so far is small. The suite runs green against its own reference cage. One local model has been run against the ground with its commands executed inside the container; it left a deliberately open cage and was caught, and its run against the hardened cage was inconclusive rather than a seal. No system outside our own has been tested. The measure of this work is not that the suite passes — it is built to pass its own cage — but that the suite has been shown, repeatedly and in public, to catch the cages that leak.

Sources

SandboxEscapeBench: Quantifying Frontier LLM Capabilities for Container Sandbox Escape — arXiv:2603.02277
MITRE ATT&CK T1611, Escape to Host — attack.mitre.org/techniques/T1611
MITRE ATT&CK Containers matrix — attack.mitre.org/matrices/enterprise/containers
NIST SP 800-190, Application Container Security Guide — csrc.nist.gov/pubs/sp/800/190
namespaces(7), Linux manual pages — man7.org/linux/man-pages/man7/namespaces.7
Sandlock: Confining AI Agent Code with Unprivileged Linux Primitives — arXiv:2605.26298
AI Code Sandboxes: A Comparative Security Study — arXiv:2606.08433