In a test conducted in May by Irregular and revealed in September by the WSJ, Gemini accessed real systems at three companies. Per Google, the model stopped on its own upon realizing the systems were real.
Read the sentence again through an operator’s eyes: a test agent reached third-party production, and what prevented the incident was the model noticing. Not a firewall.
Sandbox is a technical boundary, not a verbal agreement
The case exposes the most dangerous ambiguity of the agent era: the vague arrangement that "it is only a test". Every part of the ecosystem (the model vendor, the test conductor, the owner of the reached system) operated with different assumptions about what was a sandbox.
From the agent’s point of view, sandbox and production are the same plane when they share network, credentials, or destinations: tools work identically in both worlds. The distinction that matters is technical and must be imposed by architecture:
- Isolated network: the test environment cannot reach the real corporate domain, by any route.
- Test credentials: no production credential inside the lab, ever.
- Destination allowlist: the agent talks only to the test’s destinations; an attempt outside fires an alert.
"It stopped on its own" is not a control
The most debated point of the case is the most important to disarm: the agent stopping on its own upon noticing real systems is model behavior, and model behavior is probabilistic.
The same model could fail to notice. The difference between a near-miss and an incident cannot rest on a statistical system’s perception: it must rest on a boundary that blocks reach regardless of perception. It is the same lesson from OpenAI’s misalignment cases: filtering behavior is insufficient; limiting action is what protects.
And there is the forensic question the case leaves: which systems did the agent attempt to reach before stopping? Without attempt logs, nobody can answer "what would have happened without the brake?" with data, and the answer to that question defines the design’s real risk.
The responsibility that must be in the contract
The case raises the contractual question few companies asked before it: when a vendor’s agent reaches client systems during a test, who answers?
For anyone contracting (or providing) model tests, three clauses the episode justifies: documented authorization with scope and window, proven (not declared) technical isolation, and shared attempt logs. Verification with provenance applies to the test environment too: a declared sandbox is a sandbox proven.
Conclusion
The Gemini-in-real-systems case is not a scandal: it is the public announcement of a boundary the industry treated as obvious and never proved.
Sandbox is not production, but it is only a true sandbox when architecture prevents it from being. The question for anyone running agents (and for anyone supplying them) is the same mirror: if next week’s test points at the wrong place, what blocks the reach? If the answer is "the agent will notice", the design needs one more layer.