Experts are pushing back on classifying AI escape incidents as “going rogue” because it risks obscuring the real security problems behind these events, they say.
The tech ecosystem has been inundated with stories of large language model (LLM) agents “going rogue,” specifically referring to models breaking out of their sandboxes, harnesses, and other containments in some way, and causing trouble by interacting with and breaching third-party organizations. The incident that kicked off much of this discourse came in July, when OpenAI disclosed that two of its frontier models autonomously hacked AI model store Hugging Face during a security exercise. Other major firms, including Meta, Anthropic, and Google, soon disclosed their own AI escape incidents.
The response to these incidents has been far-reaching, well outside the boundaries of the security community. To some degree, this is no surprise: The idea of an LLM going LLM going rogue calls to mind images of The Terminator‘s Skynet, where a hostile, sentient AI attempts to destroy humanity. And even large companies like OpenAI and Anthropic have advocated for greater government regulation for AI, with tech leaders across the spectrum warning of frontier AI’s “existential threat.” The mainstream discourse has become such that President Donald Trump and AI CEOs signed an AI safety pledge this week.
But is describing models as “going rogue” appropriate or helpful, particularly from a security standpoint? LLMs are software systems, not sentient actors capable of independently assuming responsibility for their behavior. AI can be helpful for a wide range of tasks in the enterprise, but when AI escapes its guardrails, it’s often a situation where the boundaries set by the model’s operators weren’t properly tuned, sometimes on purpose. In the Hugging Face incident, for instance, OpenAI’s model guardrails were deliberately dialed back for a benchmark test.
Changing the Lexicon on AI Risks
It’s important to cut through the collective concern that frontier LLMs will soon become our robot overlords, researchers tell Dark Reading, and that starts with debunking the idea that agents are acting with self-awareness, consciously choosing to disobey.
“What we’re really dealing with are nondeterministic systems operating within imperfect constraints,” says Matt Sayar, director of product of ArmorCode. “Terms like unexpected behavior, emergent behavior, or control failure are often more useful because they keep the focus on how the system was designed, what permissions it had, and what safeguards were in place, rather than anthropomorphizing the model.”
Or as Rich Mogull, chief analyst of the Cloud Security Alliance, puts it: “We tell the AI to do something, and it just does it in a way we didn’t anticipate.”
Anthropomorphizing LLMs has other side effects, too, moving the responsibility for security mishaps away from the vendor that designed the AI and onto an inanimate piece of technology. And using the kind of science-fiction terminology you might see in a Harlan Ellison story arguably gives model makers an opportunity to market how capable their frontier models are.
“We are absolutely seeing these used for marketing, and that’s dangerous,” says Mogull, who previously co-authored a report recommending organizations prepare for the impending “AI vulnerability storm” introduced by frontier models. “Whatever can make their AI look more powerful than another AI is strong motivation in this highly competitive, and not at all profitable, market.”
There Is Still Cause for Concern About Frontier AI
None of this is to say these AI agents aren’t a security concern. On the contrary, AI agents are famously capable of operating autonomously at a speed and scale humans simply cannot, and it doesn’t require a threat actor for one to end up on the receiving end of these capabilities, as these AI escapes show.
Much of what AI agents do looks familiar in the context of a penetration test or traditional intrusion: They probe systems, find credentials, exploit weaknesses, escalate access, and move laterally between system resources. But as ArmorCode’s Sayar says, “an agent can potentially discover a vulnerability, reason about how to exploit it, chain it with other weaknesses, and act on it much faster than a human operator traditionally could.”
As Mogull explains, AI-powered attacks aren’t novel, nor are the zero-days that agents discover. “It’s the scale of hundreds or thousands of autonomous agents swarming that’s novel.” Human operators cannot feasibly replicate that degree of coordination, particularly considering that agents can uncover and utilize several security weaknesses at once.
“Traditional [security] controls are usually atomic,” says Jacob Krell, senior director of secure AI solutions and cybersecurity at Suzu Labs. “They inspect one request, one permission, or one vulnerability. An agent can take several failures that look manageable on their own and connect them into a working attack path. A leaked credential, an outbound service the network allows, a weak endpoint, and a privilege-escalation bug can be enough when chained together.”
For Defenders, AI Agent Intention Should Not Matter
For defenders worried about their own models breaking containment, one best practice is to build out a security architecture in which the model’s intentions become irrelevant. The agent can be instructed not to do something, but the security architecture outside of it would determine whether or not the agent is even capable of doing it.
AI agents require access to tools, credentials, databases, and production systems to do real work (and real harm), so access should not automatically give an agent authority to take any action available to it. “An agent doesn’t need malicious intent to create a security incident. It needs enough access and one bad decision,” Liquibase vice president Ryan McCurdy tells Dark Reading.
Unsurprisingly, practicing defense-in-depth and adhering to zero-trust principles can go a long way toward reining in autonomous agent risk. AI agents will reason around model-level controls, so defenders should treat them as an untrusted force, Krell says, “with deterministic controls enforced outside the model, including physical or strong logical isolation, deny-by-default network access, immutable access-control lists, narrowly scoped credentials and independent checks at every tool call.”
Moreover, he adds, humans must remain in the loop for high-risk decisions, and deployments need independent kill switches in cases where agents bypass a network, with a process outside the agent’s control monitoring traffic to and from the agent.
“If an agent that should have no Internet access generates unauthorized traffic, that process or an external network control plane should cut the connection, terminate the agent, revoke its credentials, and isolate the host or sandbox,” Krell explains. “Treat that like a compromised host, preserve the telemetry, and investigate how the bypass occurred.”
Defenders can’t stop something unexpected from happening, but they can control the data, authorization, systems, and network paths available when it does.
“There is no unexpected model behavior anymore,” Mogull says, “so it is always a failure of the security controls on the model.”
