OPINION
Recently, Robert Lemos reported in Dark Reading that confidence in autonomous penetration testing is falling: The share of organizations willing to rely on it dropped to 9% in 2026, down from 29% a year earlier, according to a Cobalt report. Companies are still experimenting with AI systems that hunt for weaknesses, but far fewer are leaning on them the way they did a year ago. The obvious explanation is that the technology overpromised and is now settling into a trough. I think something more specific is going on, and it carries a lesson that applies to autonomous defense just as much as offense.
When we field an autonomous security system, we wrap it in rules. We set authority ceilings so it cannot act beyond a certain level without a human. We add recovery protocols, so it steps back to a safe posture when something looks wrong, then steps forward again slowly once conditions clear. We log every decision, so the whole thing is auditable.
This is good engineering and good governance, and most of us are proud of it. The problem begins once those rules are written down, certified, and described in a standard or a product sheet. From that point on, the rules are not only your safeguards. They are public information, and a capable adversary can read them and act on what they read.
Consider a defensive system with sound governance at the edge of your network. It evaluates the evidence correctly on every cycle, stays under its authority ceiling, and records everything for review. Over eight weeks, nothing in its logs looks wrong on any single day. Yet the system drifts steadily into a permanently supervised state where a human must approve nearly every action. No one breached it, and no exploit was fired.
The adversary simply understood the recovery rule, which by design lets the system step back fast but recover slowly, and engineered a rhythm around it: short bursts of low-grade probing that knock the system down, followed by quiet periods during which it crawls back up. The probes cost the adversary almost nothing. The recovery is free, because it’s just your system behaving exactly as specified. The result is a defense that has been talked out of its own authority without a single line of its code being touched.
Cap Weaponization
I call this cap weaponization: the exploitation of the governance layer itself as an attack surface, separate from the technical attack surface of the systems it governs. The underlying move is old, even if the setting is new. Anyone who has watched a metric get gamed knows Goodhart’s law, where the moment a measure becomes a target, people optimize the measure and abandon the goal. The same dynamic drives lawfare in international conflict, where an adversary turns your own commitment to the rules into a weapon against you. AI safety researchers see it too, in specification gaming, where a model satisfies the letter of its objective while trampling the intent. Autonomous defense is the next place this same move appears, and the stakes there are about as high as they get.
The slow-drift example is only one opening. The same governance layer offers several. An adversary can feed false inputs, so the authority computation reads conditions that aren’t real. Another path is to slip past the ceiling through configuration drift rather than a clean exploit. A third move is faking the signals that say a threat has passed, so the system recovers authority it has not actually earned. And in coalition setups, the weakest participant becomes the target, since one soft node can drag the shared decision down. None of these requires breaking your cryptography or finding a zero-day. They require reading your specification and being patient.
If that sounds bleak, there is a workable answer for defenders. The fix is not to hide your governance, because hiding it forfeits the transparency that makes the system reviewable and trustworthy in the first place. The fix is to make one property impossible to fake: The authority your system acts on must always equal the authority your audit records. Compute it in one place, enforce the ceiling in a separate place, record both, and have your review process cross-check them. An adversary who manipulates one of those layers without simultaneously manipulating all of them produces a mismatch your audit can catch. That cross-layer consistency, rather than the unbreakable correctness of any single number, is what holds up under pressure.
Three things follow that a security leader can act on:
-
First, root the part of the system that enforces the authority ceiling in hardware, with its own independent record, rather than trusting software to police software.
-
Second, stop reviewing your autonomous systems one cycle at a time and start reviewing the long window. The eight-week drift in my example is invisible day-to-day and obvious over the quarter, but only if someone is comparing actual behavior against an expected operating profile.
-
Third, red-team the governance, not only the code. We already pay people to attack our technical surface before we deploy; we should pay them to attack the rulebook too, walking through exactly these moves against the policy before it ever meets a real adversary.
The decline in confidence in autonomous security tools is partly a story about models that underdeliver, and partly about adversaries learning that the cheapest way to defeat an autonomous system is often to attack the rules we built to keep it safe. The teams that stay ahead will treat their own governance with the same suspicion they already aim at their firewalls.
