LLM Penetration Testing · Moritz Samrock

LLM Penetration Testing is the authorized, methodical security assessment of systems built on Large Language Models. While AI Penetration Testing covers the broad spectrum of ML systems, LLM pentesting focuses specifically on the unique attack surface of language models: prompt injection, output manipulation, system prompt extraction, retrieval-augmented generation (RAG) poisoning, tool and function call abuse, and data exfiltration through conversational interfaces. As organizations deploy LLM-powered chatbots, copilots, and autonomous agents, these systems present novel vulnerabilities that demand specialized testing.

Who commissions this test?

Product security teams at companies shipping LLM-powered features, AI safety teams, CISOs responsible for customer-facing AI products, and engineering leaders integrating LLMs into internal tools. Organizations building chatbots, copilots, AI agents, document processing systems, or any product where an LLM processes untrusted input should commission this test. The OWASP Top 10 for LLM Applications has raised awareness across the industry, driving demand from compliance and risk management teams.

Test objectives

The goal is to identify vulnerabilities specific to LLM deployments: can an attacker override the system prompt, extract confidential instructions, manipulate the model into performing unauthorized actions through its tools, poison the knowledge base, exfiltrate data through the conversation, or bypass safety filters? Each finding is assessed for exploitability, blast radius, and business impact.

What is tested?

Testing covers the full LLM integration stack: system prompts and instruction hierarchy, input preprocessing and prompt construction pipelines, output filtering and safety mechanisms, tool and function calling implementations (what the LLM can do), RAG pipelines and knowledge base integrity, conversation memory and context window handling, rate limiting and token consumption controls, multi-turn interaction patterns, integration with downstream systems (databases, APIs, email), and the boundary between LLM-generated content and application logic.

Common findings

Typical vulnerabilities include direct prompt injection that overrides system instructions, indirect prompt injection via user-supplied documents or web content ingested by the model, system prompt leakage through carefully crafted queries, excessive tool permissions allowing an LLM agent to perform unintended actions (file access, database queries, API calls), RAG poisoning through manipulated knowledge base documents, PII leakage when the model reproduces training data or context window contents, token limit abuse enabling denial of service, insecure output handling where LLM-generated content is rendered as HTML (leading to XSS), hallucination exploitation to generate convincing but false information, insufficient output filtering allowing harmful content generation, context window manipulation to overwrite earlier instructions, and injection attacks through LLM outputs that reach downstream systems (SQL, shell commands). Testing methodology follows the OWASP Top 10 for LLM Applications.

Typical engagement workflow

An LLM penetration test follows the established pentest process with specific adaptations for language model deployments.

Interest and initial inquiry — the client describes their LLM-powered product, the model provider and version, and the business reason for testing (pre-launch, compliance, incident-driven).

Scoping discussion — testers and the client discuss the LLM architecture: which model (proprietary, open-source, fine-tuned), how it is integrated (API, self-hosted), what tools and functions the model can call, the RAG pipeline architecture, and whether the system prompt is considered confidential. This conversation determines the testing depth across prompt injection, tool abuse, RAG poisoning, and output manipulation.

Proposal and approval — a proposal outlines scope, methodology (referencing OWASP Top 10 for LLM Applications), timeline, and deliverables. LLM pentests may require iterative scoping as testers discover new attack vectors during assessment.

Scope definition — targets are precisely documented: application URLs, API endpoints, model version, available tools and functions, RAG data sources, test accounts with different permission levels, and any constraints (e.g., do not attempt to fine-tune or modify the model).

Letter of Engagement — authorizes testing and explicitly addresses whether testers may attempt system prompt extraction, tool abuse in production, and interaction volume limits to control costs (API calls to commercial LLM providers generate charges).

Additional authorizations — API access provisioning, test environment setup, and cost controls. For LLMs hosted by third-party providers, the client may need to arrange rate limit increases or dedicated test instances.

Information provisioning — depending on the approach, the client provides system prompts, tool definitions, RAG pipeline documentation, function schemas, conversation flow diagrams, or full source code. Gray-Box testing is standard: testers receive the application but not necessarily the system prompt, reflecting a realistic attacker perspective.

Kick-off call — alignment on testing approach and cost management. LLM pentests can generate significant API costs; the kick-off establishes budget limits and monitoring for token consumption.

Execution with ongoing communication — testers systematically probe the LLM deployment: attempting prompt injections across multiple attack vectors, testing tool and function call boundaries, probing RAG integrity, evaluating output filtering, and exploring multi-turn exploitation chains. Critical findings — system prompt extraction, unrestricted tool access, data exfiltration — are reported immediately.

Vulnerability collection and rating — findings are documented with exact prompts used, model responses, reproduction steps, and severity ratings. LLM-specific impact factors include the scale of potential abuse (every user could exploit the vulnerability), reputational risk from safety bypasses, and data exposure scope.

Final report — comprehensive documentation with executive summary, detailed findings organized by attack category (prompt injection, tool abuse, RAG poisoning, output handling), risk ratings, and actionable remediation recommendations covering system prompt hardening, tool permission scoping, output sanitization, and monitoring.

Presentation — results are presented to product, engineering, and security leadership. Demonstrations of successful prompt injections and tool abuse are particularly impactful for illustrating risk to non-technical stakeholders.

Project closure — the engagement concludes with remediation priorities, retest scheduling, and recommendations for ongoing LLM security monitoring. Given that model updates, system prompt changes, and RAG knowledge base updates each introduce new risk, periodic retesting is essential.

Who should commission this test — and when?

Any organization deploying LLM-powered features should commission this test before launching LLM-based products, after system prompt or tool definition changes, after RAG knowledge base updates, after switching or updating the underlying model, for EU AI Act compliance (high-risk classification), and following any LLM-related security incident or public jailbreak disclosure. Organizations with customer-facing LLM agents should test at least quarterly, as the prompt injection landscape evolves continuously and new attack techniques emerge regularly.

AI Penetration Testing: The broader assessment of AI/ML systems, of which LLM testing is a specialization.
Penetration Testing: The overarching discipline of authorized security testing.
Adversarial Machine Learning: Techniques for manipulating ML models through crafted inputs.
Guardrail: Safety mechanisms constraining LLM output within acceptable boundaries.
Injection Attack: Inserting malicious instructions into a data channel to alter system behavior.