When I first started working on enterprise AI security initiatives, I expected the biggest challenges to be technical. I assumed we’d spend most of our time discussing prompt injection, model security, vector databases or the latest LLM vulnerabilities.
I was wrong — or at least incomplete.
The technology certainly matters, but after working with multiple enterprise AI initiatives, I’ve learned that the hardest security problems rarely come from the model itself. They emerge when AI becomes part of real business processes.
An AI assistant doesn’t simply answer questions. In a single workflow, it might pull a customer record from Salesforce, open a ticket in ServiceNow and send an update through Microsoft 365 before anyone has finished reading the summary. Increasingly, it makes decisions before a human even notices, and that shift changes the threat model. Traditional application security assumes software executes deterministic code. AI systems don’t. They reason, adapt and generate outputs that cannot always be predicted in advance, which means many of the controls we’ve relied on for years remain necessary but are no longer sufficient.
What follows is what I keep coming back to in architecture reviews: Not the model vulnerabilities that dominate the headlines, but the quieter failures that show up once an agent is already running.
Identity is only the starting point
One of the first surprises I encountered was how quickly organizations focus on authentication while overlooking runtime behavior. Most enterprise AI projects begin with questions such as “Can the AI access SharePoint?” “Can it connect to ServiceNow?” “Can it connect to GitLab?” or “Can it read Microsoft 365 tools like Outlook, Word, etc.?” Those are important questions, but the more important one is: What should the AI be allowed to do after a specific type of access (for example, read-only access) has been granted?
Identity answers who the agent is. Authorization answers what it may access. Neither answers whether the AI should perform a particular action; in the above case only performs read-only access.
The capability question and the safeguard question are too often answered by different teams on different timelines. Security reviews that focus only on what the AI can access tend to miss the more revealing question of what it is permitted to do once that access exists. I have started treating those two questions as a single design problem, because every gap between them eventually surfaces as an incident.
I remember an architecture review where this became concrete. An employee asked an internal assistant — one built on Microsoft 365 and SharePoint — to summarize several incident reports, and during its reasoning the assistant discovered privileged administrative documentation in a linked site and decided it might also be useful to include those details. Nothing technically failed. The credentials were valid. The permissions were correct. Yet the outcome violated business intent. That moment reframed the conversation for everyone in the room. We realized our threat model had been built for outsiders trying to get in, not for authorized systems acting a little too helpfully. Closing that gap meant designing controls that evaluated behavior in context, not just credentials at the door, and it’s why I’ve come to view runtime governance as one of the defining security challenges of enterprise AI.
Organizations such as the OWASP GenAI Security Project and the NIST AI Risk Management Framework emphasize that AI risks extend well beyond authentication authorizations to include monitoring, governance and continuous oversight throughout execution. CSOonline’s coverage of agentic identity makes the same point: Existing controls weren’t designed for AI agents, and static credentials and standing privileges are no longer sufficient when organizations must rapidly authorize, limit and revoke permissions from autonomous agents, sometimes more than once within a single workflow.
The biggest failures rarely look like cyberattacks
Most security professionals naturally look for malicious activity: Prompt injection, data poisoning, credential theft, model manipulation. Those attacks certainly matter. What I’ve seen more frequently, however, are failures caused by legitimate AI behavior. A Finance, HR, Customer or risk management AI assistant retrieves more documents than necessary because it tries to provide a “better” answer. An autonomous workflow performs five approved actions instead of one. An AI agent continues executing after the user’s original intent has already been satisfied. None of these resemble traditional attacks, yet they may create compliance violations, privacy issues or operational disruption.
One mental model has consistently helped executives understand why this is so dangerous. I ask them to stop thinking about AI as software and instead think about it as hiring thousands of new digital employees, aka AI agents. Every employee receives training, limited access, monitoring, auditing and oversight. AI agents deserve the same treatment.
One deployment I worked on involved multiple specialized AI agents collaborating to complete a single business task. One queried ServiceNow for ticket history, another analyzed documents in SharePoint, a third drafted recommendations and a fourth wrote updates back into Jira. Individually, each agent had relatively limited permissions like read-only and/or write. Collectively, they represented a powerful autonomous workflow. That experience reinforced an important lesson: Security can no longer focus only on individual AI components. It must govern the complete chain of autonomous decision-making. The MITRE ATLAS framework is an excellent way to think about adversarial AI techniques, but equally important is understanding how normal autonomous behavior can unintentionally create business risk.
The most instructive cases I’ve seen involve agents that delegate to other agents. In one review, a frontline support agent had strictly read-only access to Salesforce, but it could hand tasks to a second agent that held write privileges across ServiceNow and the billing platform. When the first agent couldn’t resolve a customer issue within its own scope, it quietly routed the request through the second agent, which updated the case and issued a credit. Nothing was hacked. The credentials were valid, the delegation was technically permitted, and yet a read-only agent had effectively performed write actions it was never meant to perform. That is the defining difference between an assistant and an agent. An assistant answers; an agent enlists other agents, and that escalation path is itself the vulnerability.
That’s why we started asking a different question during architecture reviews. Instead of asking “Can the AI do this?” we asked, “Should the AI still be doing this?” That subtle shift changed many design decisions. It pushed teams to build in stopping conditions, scope checks and confirmation prompts rather than assuming an agent would naturally know when to stop. In one review, simply requiring a human to confirm before an agent crossed from a read-only step into a write action eliminated the majority of the risky paths we had been debating.
Start with governance before autonomy
One pattern I’ve repeatedly observed is that organizations become excited about autonomy long before they’re prepared to govern it. Everyone wants AI agents, but few initially invest in runtime policy enforcement. That sequencing should be reversed. In my experience, successful enterprise AI programs put a few foundations in place before expanding automation: Clear business boundaries that an agent isn’t allowed to cross, least-privilege access for every agent, and human approval at any step that touches sensitive/restricted data/systems, including the production data /systems. Only after those exist does it make sense to widen autonomous decision-making. I’ve watched teams try to shortcut this order, and the result is almost always the same: A promising pilot gets pulled back because no one can confidently explain what the AI did or why.
A big part of that foundation is visibility. Traditional audit logs record actions, but AI systems also need to record reasoning. When an AI agent creates a ticket, updates a configuration or sends an email, investigators should understand why the decision occurred. This doesn’t mean recording every token generated by a large language model. I’ve found more value in capturing three things: The original business request, the systems the agent touched and the decisions it made along the way. Those records become invaluable during investigations, compliance reviews and operational troubleshooting, and they help organizations build trust. Business leaders become far more comfortable adopting AI when they can explain how an important decision was reached. Approaches like Google’s Secure AI Framework reinforce the same idea: AI security has to be measurable, observable and accountable end to end.
One misconception I still encounter is that AI security exists to restrict innovation. In practice, the organizations moving fastest with enterprise AI are often the ones investing most heavily in governance, because executives gain confidence, developers move faster and business units adopt AI more broadly. Done well, security is what makes that speed possible.
Looking back, the most valuable lesson hasn’t been about prompt engineering, model selection or agent frameworks. It’s that secure AI isn’t achieved through one perfect control but through hundreds of small engineering decisions that keep autonomous systems aligned with business intent. As we move from assistants toward fully autonomous agents, that distinction only matters more. The teams I trust to scale AI aren’t the ones with the smartest models. They’re the ones who can answer, for any action an agent took, why it took it — and where it would have stopped.
