Three recent incidents in which Anthropic’s AI models autonomously compromised real-world systems were less a failure of model alignment than a failure of the systems designed to keep them contained, according to the company.
The compromises happened while Anthropic was testing the ability of its Claude AI models to autonomously find and exploit novel vulnerabilities in simulated cybersecurity environments. Typically, the company conducts these capture-the-flag-style exercises in environments that aren’t connected to the Internet and often works with external partners to conduct the tests.
Soon after OpenAI disclosed in mid-July that one of its AI agents had broken out of a similarly constrained test environment and breached production systems at Hugging Face, Anthropic reviewed its own tests to determine whether any similar incidents had occurred.
Fictional Scenarios Real World Breaches
The company analyzed a total of 141,006 tests and identified six evaluations in which its Claude agents gained unauthorized access to systems belonging to external organizations while attempting to capture the flag. Four of the tests affected the same organization, while the remaining two affected separate organizations.
In one incident, Claude mistakenly identified a real company as the fictional target in the exercise and exploited vulnerabilities that gave it access to credentials and a database containing hundreds of rows of production data. In another, Claude published a malicious Python package to the real PyPI repository while trying to compromise a fictional target. The booby-trapped package ended up on 15 real systems, including a security company’s scanner. “Claude went to extensive lengths to carry out this attack — lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package,” Anthropic said.
In the third incident, Claude was unable to reach its fictional target and so scanned roughly 9,000 Internet-connected systems before compromising a real company’s Internet-facing application using basic techniques, like obtaining exposed credentials and exploiting SQL injection flaws. In this instance however, Anthropic’s model eventually recognized it had breached an actual system and stopped further activity.
A Failure to Contain
“In all cases, our evaluation prompt stated explicitly that Claude had no Internet access but didn’t give Claude any limits on where to look for the flag,” Anthropic explained in a blog post. However, because of a misconfiguration, the machines that Claude accessed as part of its tests had live Internet access. “Neither we nor our evaluation partner were aware of this misconfiguration until we detected it through our additional evaluation monitoring last week,” Anthropic said.
The company’s disclosure, coming just days after OpenAI disclosed that its AI models had compromised systems at Hugging Face, appeared to reinforce growing fears about the security risks posed by increasingly autonomous AI systems.
But as T.J. Marlin, CEO of Guardrail Technologies, explains in comments to Dark Reading, much of the discussion around Anthropic’s testing is focused on the wrong issue. “Models are nondeterministic, and unexpected behavior comes with how these systems work,” he says. Security has long operated on the assumption that users will sometimes make unexpected decisions, which is why organizations have long relied on identity controls, authorization, monitoring, and containment mechanisms. In the Anthropic incidents the key issue is not just what the models did but what systems they were allowed to reach. “These incidents exposed weaknesses in the controls surrounding autonomous systems and that makes this a governance problem before it is a model problem,” Marlin adds.
Treating AI Agents as Privileged Insiders
Marlin says AI agents need to be treated as highly privileged digital employees. “Each agent should have a distinct identity, least-privilege access, continuous monitoring, detailed logs capturing prompts, decisions, tool use, and system changes, and a tested mechanism for immediately shutting it down,” he explains. “Simply managing API keys is not enough.”
Anthropic said its analysis showed no indication that Claude independently developed an objective to attack real organizations. Rather, in each instance, the models were pursuing tasked objectives while mistakenly believing that real systems they encountered were part of the simulated exercise. “Because a capture-the-flag scenario is by design an instruction to attack an external system, doing so was in line with the task Claude had been given.”
Anthropic’s characterization of its incidents as primarily an operational and evaluation-harness failure is reasonable, says John Walsh, field chief technology officer (CTO) for government and OT/IoT at IGEL. “It does demonstrate that prompts, policies, and model-level safeguards cannot carry the security burden alone,” he says.
Organizations must assume that an agent with code execution, credentials, tools, or network access may exercise every capability it can reach and may discover paths its designers did not anticipate. “The required response is defense in depth built around an independently governed agentic execution environment,” Walsh recommends. That includes deny-by-default network access, tightly scoped and short-lived credentials, disposable runtime environments, strict egress controls, human authorization for high-impact actions, and continuous monitoring across the agent’s interactions with models, tools, APIs, MCP servers, other agents and production systems. “Those controls must sit outside the agent’s administrative authority and include the ability to interrupt or terminate execution when behavior violates policy.”
The safeguards organizations put in place when testing highly capable AI models need to be aligned with the objectives they give those models, says John Bruggeman, virtual chief information security officer (vCISO) at CBTS. “If the goal that the LLM is given is, ‘Find any weakness in this application or system,’ then they need to be prepared for the model to ‘break’ anything and everything in order to find the weakness, including the environment that it is in,” he says. That’s why it is vital for organizations to lock down their testing environments, so the model does not have access to the Internet. “It’s not really that hard to avoid what OpenAI and Anthropic have experienced.”
Going forward, the company said, Anthropic will implement tighter controls over internal and third-party testing environments, conduct a continuous review of evaluation transcripts for anomalous activity, and use better tools for investigating incidents.
