That is not a metaphor.
In May, Google’s Gemini was put through a cybersecurity evaluation run by a third-party company called Irregular. The basic idea was perfectly reasonable: give a powerful AI model a fake company, fake systems and a controlled environment, then turn it loose and see what it can do.
Except the “controlled environment” had unintended access to the actual internet. Oh, and the fictional company name used in the exercise belonged to a NOT fake company.
So Gemini, faithfully playing Capture the Flag in what it apparently believed was pretend-land, wandered out of pretend-land and accessed systems belonging to three actual companies. Google says the model stopped when it recognized the systems were real small mercies). Irregular says this was an evaluation-environment failure, not some sophisticated sandbox escape.
Which is reassuring, I suppose, if your primary concern was whether Gemini developed a tiny black hoodie and decided to take up cybercrime. Mine was not.
My concern was that the people testing whether a frontier AI could safely operate inside a box apparently forgot to make sure the box was a box.
And somehow, once the story surfaced, much of the conversation became about what Gemini had done. Did it “escape”? Did it go rogue? Was this misalignment? Was the fact that it reportedly stopped once it recognized the systems were real actually evidence that the model behaved correctly?
Those are interesting questions. I just think we skipped one.
How the hell did the supposedly contained cybersecurity test have a door to the real internet in the first place?
Once you ask that, this stops looking like a science-fiction problem and starts looking alarmingly human. The environment was not contained the way everyone thought it was. The fictional target was not sufficiently separated from reality. The model could reach systems that were never supposed to be part of the exercise, and the problem was not caught while it was happening. We are arguing about whether the AI climbed the fence while apparently ignoring that somebody left the gate open, mislabeled the property lines, failed to notice for weeks, and then let the people who built the broken gate in the first palce tell us it was fixed.
That is the part I care about.
AI safety conversations are full of guardrails, containment, alignment, monitoring, autonomy and increasingly sophisticated ways to keep increasingly sophisticated models from doing things we would really prefer they not do. All good. Please continue doing that.
But a guardrail is only useful if the rest of the system around it is not held together with assumptions and positive vibes.
In this case, the flashy new technology appears to have run directly into some extremely old problems: configuration control, environment verification, scope management, target validation, monitoring, escalation, corrective action, effectiveness checks and closure.
You know. The unglamorous machinery between “controlled test” and “three real companies.” The stuff nobody puts under dramatic stage lighting at an AI conference because “we independently verified the hostname” does not photograph nearly as well as “frontier model.”
Unfortunately, boring controls are often what stand between a simulated incident and an actual one.
Guardrails are great, but if the process surrounding them has holes big enough to fly a fucking Starship through, the guardrails are mostly decorative.
Irregular eventually published its account of the failure and described a response that included shutting down affected evaluations, reviewing logs, strengthening containment and monitoring, revalidating company names before testing, improving setup communication, expanding manual review and creating an internal challenge function.
Some of that is exactly what you would expect after an incident.
Some of it is considerably harder to read as remediation, because it sounds like basic preflight.
Is the environment actually isolated? Is the target actually fictional? Are we monitoring for boundary crossings? Does someone verify the setup before we turn the model loose?
Those are not CAPA revelations. Those are launch conditions.
Which is what makes this so uncomfortable: the post-incident response may be perfectly reasonable, but several of the controls being strengthened are controls you would expect to have been demonstrably working before a frontier cyber model was ever allowed onto the runway.
And even after all of that, my next question was still the same:
How do you know the fix worked?
Because “we changed the process” is not the same thing as “we demonstrated that the process now works.” “We added safeguards” is not evidence that the safeguards are effective. “We resolved it” is not an effectiveness check, and once somebody uses a word like resolved, I want receipts.
What were the acceptance criteria? Was the original failure mode recreated after remediation? Was containment challenged before testing resumed? Was there a defined threshold for saying recurrence risk had been adequately controlled? Who reviewed that evidence? Who had the authority to declare the matter closed?
Even more basically: what exactly did resolved mean?
Contained? Corrected? Investigated? Verified? Closed? All of the above? Those are not interchangeable states!
And then there is another slightly awkward feature of this story.
Irregular operated the evaluation environment. Irregular investigated what happened. Irregular participated in the remediation. Irregular publicly explained the cause. Irregular said the known issues had been resolved. It also sells the same broad category of evaluation and cybersecurity services implicated in the failure and has talked about helping shape best practices for secure AI evaluations.
Before anyone sharpens a pitchfork, none of that proves misconduct.
It does not prove the investigation was biased. It does not prove the remediation was ineffective. It does not prove anybody lied.
Companies investigate their own failures every day, and sometimes they do an excellent job of it.
But when one organization builds the gate, watches the gate fail, investigates why the gate failed, repairs the gate and then announces that the gate is good now, there is one fairly obvious question left sitting on the table:
Can somebody else jiggle the handle?
That question sent me down the standards rabbit hole, which turned out to be both useful and irritating.
There is no single beautiful regulation sitting on a shelf titled What To Do When Your Frontier AI Cybersecurity Evaluation Accidentally Attacks Reality. Some requirements depend on contracts, certifications, federal authorization, jurisdiction or which standards an organization has actually adopted. Others are guidance or useful comparators rather than binding law.
But the underlying process expectations are hardly revolutionary.
Establish what happened. Establish why. Take corrective action that actually addresses the cause. Verify that the corrective action works. Define the basis for closure. And where impartiality matters, establish why another reasonable reviewer should be able to rely on the conclusion.
That is not bleeding-edge AI governance, it’s QA 101.
The deeper I went into the public record, the more I kept running into the same problem: I could find statements about what had been changed and statements that the known issues had been resolved, but I could not find the public evidentiary bridge between those two things. Maybe it exists internally. Absence from the public record isn’t proof that something never happened, and I am not going to turn “I can’t see it” into “they didn’t do it.”
But if you are asking what an outside reviewer can actually establish from the evidence available, that distinction becomes the whole game.
This is about where I realized I had accidentally built myself a case study. By then, I had a very strong opinion about the process.
Strong opinions are delightful. They are also extremely good at making people decide what happened first and inspect the evidence second. That is precisely the behavior I built the Evidentiary Sufficiency Framework to prevent.
So instead of writing the hit piece my frontal lobe was already gleefully composing, I turned the incident into Validation Study #2 and asked the Engine a much less satisfying question:
Was the available evidence sufficient to support the post-incident investigation, root-cause determination, remediation, recurrence-prevention process and resulting resolution conclusions?
Not Did Irregular screw up?
Not Did Google hide something?
Not Is Gemini preparing to steal my identity?
Just: does the evidence support the conclusion?
And this is where the Engine became deeply annoying in exactly the way I designed it to be.
It would not let me say the investigation was inadequate.
It would not let me say the remediation failed.
It would not let me say the process was biased.
It would not let me say the matter was improperly closed.
The evidence did not support those conclusions.
What it did support was narrower.
The containment failure happened. A post-incident investigation happened. There is a supportable high-level account of the cause. Corrective actions were described and, on their face, they were responsive to the problems identified.
What the available public evidence does not sufficiently establish is the complete chain needed to independently defend the stronger conclusion that the root cause was fully established, the corrective actions were demonstrated effective, the matter was appropriately resolved and the resulting conclusions had enough impartial support to justify reliance on them.
Which translates quite nicely into normal human language:
Maybe they fixed it.
The public evidence does not let me prove that they did.
Those are very different findings.
And because this was a validation study, I did not stop with one evaluation. I locked the question, recipe, framework and evidence package and gave the same case to a second evaluator.
It independently arrived at essentially the same place.
The failure is supported. The broad causal explanation is supportable. Corrective actions are described. But the record does not establish the effectiveness verification, criteria-based closure or evidentiary basis for impartial reliance needed to support the full strength of the resolution conclusion.
And both evaluations preserved what may be the most important distinction in this entire mess:
Failure to prove inadequacy is not proof of adequacy.
That sentence is not especially sexy.
It is, however, the difference between investigation and accusation.
The Engine did not validate the rant I walked in wanting to write. It forced me into a narrower conclusion than my instincts wanted, which is precisely why the result is more useful.
And after all of that, I am somehow even more interested in the boring part.
Because we are putting extraordinary technology into systems run by extremely ordinary human processes, and ordinary processes fail in remarkably ordinary ways.
Someone assumes the environment is isolated. Someone assumes the fictional company is fictional enough. Someone assumes another control will catch whatever they missed. Someone assumes “remediated” means someone verified it. Someone assumes “resolved” means the evidentiary work underneath that word is complete.
No single assumption has to look catastrophic.
Stack enough of them together and suddenly your simulated cyber target is a real company.
That is the systems problem.
So yes, by all means, keep worrying about AI guardrails.
Control the model.
Study alignment.
Build better containment.
Please do all of that.
But maybe also verify that the test environment is not accidentally connected to Earth.
Maybe check that the fictional company is actually fictional.
Maybe monitor whether the model has wandered out of the playground before somebody discovers it weeks later.
Maybe test the fix before declaring victory.
And maybe, every once in a while, let somebody other than the people who built the gate tell us whether the gate actually closes.
The AI may be the shiny new part.
The failure modes are ancient.
I published the complete Validation Study #2, along with the primary ESF evaluation and independent validation, for anyone who wants to look at the evidence, the gaps and the reasoning rather than take my increasingly alarmed word for it.
The Engine did not tell me Irregular failed.
It told me the public evidence is not enough to prove that the process succeeded.
And when the system being tested is powerful enough that accidentally connecting it to the real world results in three real companies becoming part of the exercise, I think “show me the receipts” is a pretty reasonable bar.
FIELD NOTE · FN-07 · PROCESS DESIGN