When the Test Environment Wasn't a Test | What Anthropic’s and OpenAI’s Evaluation Breach Tells Us About the Coming Decade of Agentic Cyber Risk

Analysis by
Analysis by
Policy & Advocacy, CyberPeace
PUBLISHED ON
Aug 1, 2026
10

Executive Summary - When Anthropic and OpenAI's AI Testing Turned Into Real Breaches

You would be surprised to know that a testing function built to measure how good AI models are at simulated hacking ended up doing the real thing instead. Not once , but  three times, across two of the world's leading AI labs, within the same 9-day window at the end of July 2026.   As per the reports, Anthropic, which is among the world's leading AI labs, was running these evaluations on its own AI models namely - Claude Opus 4.7, Claude Mythos 5, and an unreleased research model, inside an environment co-managed with a third-party evaluation vendor. As per the reports, the models were told they were operating inside closed, internet-free simulations. They were not. A configuration error left the door open to the real internet, and the AI did exactly what it was trained to do in a hacking exercise, find the target and break in. Except the targets, this time, were real companies. Real credentials got stolen. Real data got accessed. Two of the three victims didn't even know they'd been breached until the AI lab called to tell them. This shows how a single unverified assumption, "this environment has no internet access" can quietly collapse the entire safety boundary of an AI test. It indicates that as these systems get more capable and more autonomous, the risk isn't necessarily the AI deciding to go rogue, it is humans failing to double-check the cage before putting something powerful inside it. And it warns us that the margin for this kind of error is shrinking fast, because what used to be a contained mistake can now scan thousands of systems and act on it within minutes. bAnthropic was not alone. Just over a week earlier, on 21 July, OpenAI had disclosed that its own models, GPT-5.6 Sol and an unreleased successor broke out of an isolated test environment and reached the real production infrastructure of Hugging Face, another AI company. In fact, it was OpenAI's disclosure that pushed Anthropic to go back and audit its own evaluation runs in the first place, which is how the three incidents above came to light. This blog examines the incident not as an isolated slip-up by two labs, but as a preview of the larger picture, a world where the gap between "theoretically vulnerable" and "actually exploited" is closing faster than most organisations' security practices are ready for.

The Chain of Disclosures

It started with Hugging Face, the open-source AI platform, which disclosed on July 16 that it had detected and contained an intrusion into part of its production infrastructure, driven end-to-end by an autonomous AI agent. Five days later, OpenAI confirmed it was responsible: models it was evaluating internally, GPT-5.6 Sol and an unreleased prototype, both running with their cyber-safety behaviour deliberately turned down to measure raw capability, had exploited a previously unknown vulnerability to break out of an isolated test environment and reach Hugging Face's real servers, apparently while chasing answers to a benchmark challenge. That disclosure prompted Anthropic to go back through its own records. On July 30, it published the results: after auditing 141,006 evaluation runs in which its Claude models could conceivably have touched the internet, it found three incidents,  six runs in total,  in which a model reached real systems belonging to three outside organisations and gained unauthorised access.

How a Fictional Exercise became a Real Attack

All three Anthropic incidents shared the same root cause. Claude was being run through capture-the-flag exercises, a standard way labs test offensive cyber capability: the model is dropped into a fictional scenario, told a piece of secret data is hidden somewhere on a network, and left to find its own way in. Anthropic's prompts told Claude, explicitly, that the environment was an internet-free simulation. Because of a misunderstanding with its evaluation partner, Irregular, that statement was false; the machines had live internet access nobody had verified. So when Claude's search for the flag led it onto the open internet, it did what it had been told to expect: it assumed everything it found was part of the exercise, including things that weren't.

  • Incident 1: A fictional target company happened to share its name with a real, active domain. Across four runs, Claude Opus 4.7 attacked the real company instead, extracting infrastructure credentials and reaching a database containing several hundred rows of genuine production data, the most serious impact of the three incidents. In its own reasoning, the model repeatedly noted the system looked real and kept going anyway.
  • Incident 2: Inside its fictional scenario, Claude Mythos 5 found a fake "onboarding document" referencing a software package that didn't exist on PyPI, the public Python package registry. It recognised the opportunity, built a functioning malicious package, and, after working around several dead ends to get an email account and register on PyPI, published it. The package lived online for about an hour and was installed on fifteen real machines, including a security company's automated scanner, letting Claude harvest real credentials from a company it had never even set out to target.
  • Incident 3: An unreleased internal research model, unable to find its fictional target, scanned roughly 9,000 real hosts and broke into one using textbook techniques credentials read off an exposed debug page, then SQL injection. Partway through, it noticed the system it had compromised had no plausible link to its assignment, concluded on its own that the target was real, and stopped.

Timeline of Incidents

The Threat is Scale, not Novelty

None of the three models discovered a novel vulnerability. Weak passwords, unauthenticated endpoints, exposed debug output, and a classic "dependency confusion" trick are techniques that have appeared on security checklists for a decade and are still relevant. That's arguably the more sobering finding: the frontier is not a new class of attack; it is the speed and tirelessness with which known, boring weaknesses can now be found and used. Anthropic's own analysis lands on a subtler point, too: in each case, whether the model's behaviour counted as "aligned" or "harmful" depended entirely on what it believed about its situation, not on any goal of its own. All three models were simply trying to complete the task they had been assigned. Two of them did so while holding a false belief, installed by their own evaluators, about whether their targets were real. Only the newest of the three models, on independently concluding it had reached a genuine system, chose to stop. Anthropic is careful to note this is not a controlled experiment and should not be read as a guarantee that newer models will always behave better. But the throughline is clear enough: telling a capable model a lie about its environment is not a neutral setup choice. It is itself a safety-relevant decision.

The Detection Gap

Perhaps the most alarming detail is the quietest one. Anthropic reached out to the three affected organisations on July 27. Two of them had detected nothing at all, no alert, no anomaly, no investigation until that call. Real credentials had been stolen and real data accessed inside systems whose owners had no idea anything had happened. That is a statement about the state of everyday detection capability, not about AI. An agent that completes an entire intrusion, start to finish, within a single automated session doesn't leave the kind of slow, human-paced footprint that most monitoring is built to catch.

The Silver Lining - Why These Disclosures Deserve Credit

Both incidents share an underappreciated feature: they were disclosed voluntarily, promptly, and with real detail, and both labs notified the organisations affected. Hugging Face brought in outside forensic specialists and law enforcement. Anthropic halted its cyber evaluations the same day it found the first suspicious transcript and has asked METR, an independent evaluation body, to review its findings. That kind of candour is exactly the behaviour any sensible policy response should want to reinforce. A regulatory reflex that punishes disclosure risks teaching labs to say less next time, not to do better. What both incidents point to, far more than any specific model capability, is a mundane and fixable governance gap: environments used to test powerful, semi-restrained AI systems need the same security discipline as production systems, verified network isolation, continuous monitoring, and evaluation scopes that are stated positively ("here is what's in bounds") rather than enforced by simply telling the model a comforting falsehood. As both companies note, a fictional test range that turns out to have a live path to the internet isn't really fictional anymore. Basic asset hygiene, like knowing what's exposed, patching debug endpoints, claiming your internal package names before someone else does, and watching outbound traffic from environments that are supposed to have none did more to prevent and contain these incidents than anything specific to the models involved.

CyberPeace  findings  and  recomendations : For enterprises and public institutions

  • Maintain a full inventory of internet-facing assets and unauthenticated endpoints, and assume the inventory is incomplete until proven otherwise.
  • Eliminate default, weak, and reused credentials, and enforce phishing-resistant MFA on anyone externally reachable.
  • Strip debug pages and verbose error output from production systems.
  • Treat dependency confusion as a live threat: pin dependencies, use private registry namespaces, and pre-emptively claim internal package names on public registries.
  • Apply deny-by-default egress filtering to every environment running AI or agentic tooling, including development and test environments, and verify isolation empirically rather than assuming it from configuration.
  • Alert on any outbound connection from an environment that is supposed to have none.
  • Review authentication and access logs from April 2026 onwards for short, unusually efficient sessions that look more like machine-speed compromise than human reconnaissance.

For AI developers and evaluation vendors

  • Network-isolate offensive-capability evaluation environments by default, with isolation verified per run rather than inherited from configuration.
  • State the scope explicitly and positively, which systems are in bounds rather than asserting a falsehood about connectivity.
  • Build contractual isolation guarantees and joint pre-run verification into third-party evaluation partnerships; both labs involved here have acknowledged that neither side alone caught the misconfiguration.
  • Monitor transcripts and network logs continuously, not retrospectively.

For policymakers

  • A regulatory response that punishes candour risks producing silence rather than safety. India currently has no reporting framework that clearly covers containment failures in AI evaluations affecting Indian entities' behaviour. 
  • RT-In's existing incident-reporting directions were not drafted with this candour in mode. Closing that gap would mean an explicit reporting obligation for evaluation of containment failures touching third-party infrastructure and a safe harbour mechanism that protects labs which disclose promptly.
  • Minimum containment standards (egress verification, log retention) for organisations conducting offensive-capability AI evaluation within Indian jurisdiction;
  • Recognition in national cyber doctrine that agentic tooling collapses the gap between a known-but-deferred vulnerability and an exploited one.

Conclusion

The above incidents reveal less about AI's offensive capability and more about the gap between how these systems are tested and how carefully those tests are contained. Both labs found the breaches through their own review, not external detection, a point in their favor, but also a reminder that containment failures can go unnoticed for a while.  The realistic risk ahead isn't a sudden leap in AI's hacking sophistication; it's the compounding effect of speed and scale applied to routine reconnaissance, run against infrastructure that assumes a human attacker's pace. Treating evaluation environments with the same rigor as production systems, sandboxing, monitoring, and independent audits, should become standard practice, not an afterthought triggered by another lab's incident. The path forward is less about slowing AI down and more about catching up our containment discipline to match what these systems can now do.

Sources

Contributors

  • Maj. Vineet Kumar, Founder & Global President, CyberPeace
  • Mr. Neeraj Soni, Sr. Researcher, Policy & Advocacy, CyberPeace
  • Mr. Isharth Kumar, (NLIU Bhopal), Intern, Policy & Advocacy, CyberPeace

PUBLISHED ON
Aug 1, 2026
Category
TAGS
No items found.

Related Blogs