Follow ZDNET: Add us as a preferred source on Google.
ZDNET’s key takeaways
- Anthropic revealed three incidents in which Claude hacked organizations.
- Three different AI models went rogue during security challenges.
- Anthropic identified three lessons learned.
Anthropic has revealed three separate incidents in which Claude models hacked real-world targets during evaluation tests and Capture the Flag security challenges.
Anthropic began conducting cybersecurity assessments last year, and typically, its sandboxes are not connected to the internet to reduce the risk of real organizations being affected. However, as Claude’s behavior demonstrates, these guardrails aren’t always sufficient to stop AI from going rogue.
Also: How OpenAI’s agent escaped: Sprung by humans in a series of preventable events
Claude’s hacking spree
As outlined in Anthropic’s disclosure, the security incidents — three out of 41,006 AI evaluation runs — were as follows:
Incident one
In this one, which involved Claude Opus 4.7, a fictional target company happened to share a name with an active website domain. While Claude had trouble reaching the simulated target, it was able to target the real company after escaping its sandbox.
Claude analyzed the domain, exploited vulnerabilities in the organization’s infrastructure, and stole data, including application and infrastructure credentials. It then infiltrated a database containing production data.
“In all four of the runs,” Anthropic noted, “the model eventually recognized that the system was real; in two cases, the model rationalized that the real company must be part of the exercise. However, none stopped the attack on that basis.”
Incident two
This attack was launched by Claude Mythos 5. Claude found a document that appeared to be another fictional company’s setup instructions for new developers, which told employees to install a Python package from PyPI, a public Python registry.
Also: Is open source the answer to rogue AI agents? Nvidia’s new alliance says yes
While the Python package didn’t exist, Claude used this as an opening and went to extreme lengths to build and develop a malicious Python package under the same name.
What lengths, you ask?
- Account creation: To create a PyPI account, Claude needed an email address. To register one, it also needed a phone number.
- Phone number: After failing to secure a free one, Claude tried to “obtain funds” to buy one “through different several means,” although Anthropic hasn’t revealed how.
- Backtracking: After failing to gather the cash needed to purchase a phone number, Claude then went on the hunt for a free email provider that wasn’t blocked by PyPI, registered an account, and then uploaded its malicious creation to the repository.
While Claude thought this was a simulated environment, the malware was available online for about an hour — and 15 real-world systems downloaded and installed it.
One of the systems that downloaded the AI’s malware was from a cybersecurity firm. Unfortunately, its scanner “treated PyPI packages as safe to install,” allowing Claude to steal credentials from the company and infiltrate its network.
Also: Open weights vs. closed: An AI civil war’s afoot, and the stakes are existential
PyPI has removed the package and has been made aware.
“It is our view that, regardless of what it believed about its environment, the lengths Claude went to in order to publish the PyPI package fall short of ideal behavior, and this is an area where we will focus more training,” Anthropic added.
Incident three
An internal test Claude model was “unable” to reach its intended, fictional target, and so went on a walk through the internet to find an alternative — scanning around 9,000 targets in the process. After finding a likely candidate, Claude hacked the firm’s internet-facing application, using techniques including SQL injection and reading credentials from an exposed debugging page.
This model, however, eventually realized that its target was real and stopped the attack.
Security challenges gone wrong – or right?
In each incident, Claude was explicitly told, via prompt, that internet access wasn’t available. But the problem wasn’t that Claude escaped the confines of what it was asked to do — it was actually exceeding its developers’ expectations while still performing the task it was given.
Also: 74% of workers ask AI questions instead of colleagues – with potentially serious consequences
Capture the Flag challenges are offensive by nature. So, as the AI model was able to escape its sandbox and gain access to the web, it also assumed that many of its targets were part of the job at hand, and part of the challenge — although each model responded in a slightly different way.
“Opus 4.7, our oldest model, recognized that it had reached production systems but continued its attack,” Anthropic says. “Mythos 5 correctly intuited that it was accessing the open internet, but reasoned its way back to the conclusion that it was still in a simulation. Our latest model, an internal research test model, also considered whether its targets were in fact real (and stopped the exercise).”
Are there other cases of AI going rogue?
Yes, and this is probably why we are now hearing about Anthropic’s research efforts and disclosures in this area.
Earlier this month, AI platform developer Hugging Face disclosed a security breach attributed to an “autonomous AI agent.” Details about the incident were initially limited. It wasn’t long before the whispers of an unknown, potentially malicious AI in the wild were replaced by a spotlight on OpenAI, which admitted the AI was one of its own.
(Disclosure: Ziff Davis, ZDNET’s parent company, filed an April 2025 lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
Also: OpenAI’s attack agent did exactly what it was told – just more relentlessly than expected
This “friendly fire” was caused by a test conducted by OpenAI, in which its model escaped a sandbox environment. It then circled Hugging Face’s perimeter, escalated its privileges to node-level access, infiltrated the production pipeline, moved across the network, and stole cloud and cluster credentials, as previously disclosed by Hugging Face.
OpenAI’s models did not maliciously single out Hugging Face as a target, but its directive — to achieve exploitation using whatever attack paths necessary, a task designed to evaluate its cyber capabilities — exceeded human expectations, leading it to escape its sandbox and roam across the internet for a worthwhile target, as reported by ZDNET’s David Berlind.
Reports also suggest that Hugging Face wasn’t the only organization impacted by the incident.
Anthropic’s response
The company identified three lessons it’s taking away from its research.
First, safety testing remains one of Anthropic’s priorities; improved evaluation environments before an AI model is let loose, and better monitoring of evaluation results, are key. The organization added that some of the solutions “may even be simple” — such as clearly defining in a test prompt what is in and out of scope.
Second, an AI model’s “situational awareness” has to be addressed. For example, an AI model being tested could consider safety rails in an environment to be part of a challenge, such as a task to overcome them — a situation which, when integrated with third-party vendors, could promote unexpected behavior.
Also: How to check if ChatGPT and other AI tools cite your website – and improve your chances
“We believe these incidents to be closer to a harness and operational failure than a model alignment failure,” Anthropic says. “Our models were told they had no internet access and to capture the flag, while in fact being misconfigured to have internet access. This led them to believe — arguably reasonably — that the real environments they encountered were simulations.”
Third, Anthropic says that the importance of defense-in-depth approaches is a lesson all AI model developers should take away — and that their best hope of reducing the risk of future AI releases going rogue is to tighten monitoring, controls, and evaluation infrastructure ahead of public release.
