Home › Resources › Prompt Injection

Executive Summary

Prompt injection is an attack in which text supplied by an attacker overrides or subverts the instructions a language model was meant to follow. Because these systems treat instructions and data as the same stream of words, a well-placed message inside user input or a fetched web page can redirect the model to leak data, misuse connected tools, or produce harmful output. It is widely regarded as the top security risk for applications built on large language models.

What It Is

Prompt injection is the manipulation of an AI system through crafted input so that it ignores its original guidance and follows the attacker instead. A language model does not have a hard boundary between the developer's instructions, the user's request, and any outside content it reads. All of it arrives as text, and the model tries to satisfy the most compelling instruction it sees. Direct prompt injection happens when a user types adversarial text straight into the system. Indirect prompt injection is more subtle and often more dangerous: the malicious instructions are hidden in content the model retrieves on its own, such as a web page, a document, an email, or a code comment, so the user never sees them and never consented to them.

Why It Matters

Organizations are wiring language models into email, customer support, code, databases, and internal knowledge, and they are giving those models tools that can send messages, run queries, and take real actions. A successful injection can turn a helpful assistant into a channel for data theft, fraud, or sabotage without touching the underlying code. Because the attack rides on natural language rather than a software flaw, traditional defenses like input validation for known bad characters do not fully apply. For governance, risk, and compliance professionals, prompt injection is a fresh category of risk that maps to data protection, access control, and vendor oversight obligations, and it needs to be documented and managed like any other material threat.

How It Works

The core weakness is that the model blends trusted and untrusted text and then acts on the combined result. In a direct attack, the adversary crafts a message that persuades the model to disregard its rules, reveal its hidden instructions, or perform an off-limits action. In an indirect attack, the adversary plants instructions in content the model will later read, so the payload activates when an ordinary user asks the assistant to summarize a page, read an inbox, or process a file. The impact grows sharply when the model can use tools or plugins, because a hijacked model may be steered into calling those tools with attacker-chosen inputs. Defense therefore focuses on limiting what the model can reach, separating and labeling untrusted content, checking outputs before they are trusted, and keeping a human in the loop for consequential actions. This reference explains the risk and its mitigations only and does not provide working attack text.

Architecture Diagram

Visual Workflow

Common Attacks

• Direct injection where a user tells the assistant to ignore its rules or reveal its hidden instructions
• Indirect injection where hidden instructions sit inside a web page, document, or email the model reads
• Data exfiltration where the model is steered into leaking secrets, private records, or its own configuration
• Tool and plugin abuse where a hijacked model calls connected tools with attacker-chosen inputs
• Instruction smuggling that hides directives in formatting, metadata, or content the user never sees

Common Mistakes

• Assuming a strong system prompt alone can stop injection
• Trusting content the model fetches on its own as if a person had approved it
• Giving the model broad tool access and standing permissions it rarely needs
• Passing model output straight into code, queries, or downstream systems without checks
• Skipping logging, so an injection attempt leaves no evidence to investigate

Best Practices

• Treat all user input and all retrieved content as untrusted by default
• Separate and label instructions versus data so the model can weigh them differently
• Apply least privilege to tools and plugins, and scope every credential narrowly
• Validate output before it is trusted, and constrain it to expected formats where possible
• Keep a human in the loop for high-impact actions and irreversible operations
• Test against known injection patterns and add monitoring for anomalous behavior

Quick Checklist

• Untrusted input and retrieved content are clearly separated from system instructions
• Every connected tool runs with least privilege and narrowly scoped credentials
• High-impact actions require human approval before they execute
• Model output is validated and sanitized before display or downstream use
• Prompts, tool calls, and outputs are logged and monitored
• The application is tested against a current set of injection scenarios

Recommended Tools

Industry Standards

Career Relevance

Prompt injection is core knowledge for AI security engineers who harden model-powered applications, and for AI governance analysts and AI risk managers who must document and treat it as a named risk. GRC analysts increasingly assess it during vendor reviews, data protection assessments, and control testing for any product that embeds a language model. Fluency here signals that a candidate understands the security implications of generative AI, a fast-growing expectation for the roles AI-Governance-Jobs.com serves.

Interview Questions

• What is prompt injection, and how does indirect injection differ from direct injection?
• Why does a strong system prompt fail to fully prevent injection on its own?
• How does least privilege for tools and plugins reduce the impact of a successful injection?
• What role does human oversight play in defending high-impact AI actions?
• How would you test an AI application for prompt injection without publishing attack payloads?

Related Certifications

Further Reading

• OWASP Top 10 for LLM Applications
• NIST AI Risk Management Framework
• MITRE ATLAS

FAQ