- Require outputs to be grounded in authoritative sources and verify claims before acting.
- Introduce approval workflows and system checks.
- Log claims, evidence, and outcomes, and test adversarial scenarios. Regularly test workflows against misleading scenarios.
- Apply least privilege, sandboxing, and rate limits to limit the potential impact of any incident.
8. Hidden context exposure
Hidden prompt exposure expands the previous category of system prompt leakage to include hidden-context scope problems.
System prompts are starting instructions given to LLMs to guide their conversations, and can contain sensitive instructions, operational parameters, security controls, business logic, and private corporate information. Enterprises may incorrectly assume these system prompts are kept confidential, but they could be exposed.
According to OWASP, the problem isn’t that attackers can get their hands on this system prompt — the problem is that companies are putting sensitive information into this prompt to begin with, including API keys and authentication details.
Developers should operate under the assumption that hidden context is potentially discoverable.
Key preventative measures include:
- Store sensitive information such as API keys, authentication details, and database information separate from system prompts, in systems the model cannot directly access.
- Avoid relying on system prompts for model behavior control; instead implement these controls, such as detecting harmful content, in external systems.
- Deploy guardrails outside the LLM to inspect model outputs to ensure the model acts as expectated.
- Implement critical security controls such as privilege separation and authorization checks independently from the LLM in a deterministic, auditable manner.
- If a workflow involves multiple tasks requiring different levels of access, use multiple agents instead, each configured with the least privileges necessary.
9. Vector and embedding weaknesses
Companies often augment LLMs with vector databases and retrieval-augmented generation (RAG) systems, pulling relevant, up-to-date information from corporate data stores before prompts are sent off to the LLMs.
In such context, attackers may be able to trick the system into retrieving information they should not have access to, or go after these data sources directly, poisoning the model and making it give incorrect information.
According to OWASP, vector and embedding weaknesses are failures in the retrieval substrate of RAG-style systems rather than in the LLM or prompt itself. Embeddings and vector stores offer a distinct attack surface, OWASP says, because they often lack the same access controls and auditability as traditional data stores.
For example, multi-tenant or poorly filtered vector stores can return documents across tenants or beyond a user’s authorization, potentially leaking data to attackers.
Key preventative measures include:
- Implement fine-grained access controls and permission-aware vector and embedding stores with strict partitioning of datasets to prevent users from leveraging the LLM to get access to information they shouldn’t.
- Create strong data validation pipelines that only accept and process information from trusted, verified sources. For user-submitted content, such as resumes, use text extraction tools that detect and flag hidden text.
- Thoroughly review and classify combined datasets to prevent data mismatch errors and control access levels.
10. Improper output handling
Improper output handling refers to insufficient validation, sanitization, and handling of the outputs generated by LLMs before they are passed downstream to other components and systems. Because LLM-generated content can be controlled by prompt input, this behavior is similar to providing users indirect access to additional functionality.
For example, if the LLM’s output is sent directly into a system shell or similar function, it can result in remote code execution.
The category has been expanded to include insecure code that assistants generate at scale.
Preventative measures include:
- Treat the model as any other user, adopting a zero-trust approach, and apply proper input validation on responses coming from the model to backend functions.
- Follow the OWASP ASVS (Application Security Verification Standard) guidelines to ensure effective input validation and sanitization and encode the output to mitigate undesired code execution.
