A year ago, AI red teaming was mostly something model developers did. Today, most organisations deploying LLM features (support assistants, internal copilots, document processing and agents) need to test them adversarially too. Guidance has matured: OWASP's Top 10 for LLM Applications is on version 1.1, MITRE ATLAS catalogues adversarial techniques against AI systems, and last month NIST published its Generative AI Profile (AI 600-1) for the AI Risk Management Framework, which includes red-teaming among its recommended practices.
This article sets out the methodology we use to red-team LLM applications.
Step 1: scope the whole application
The model is only one component. Scope should cover:
- Entry points: chat UI, API, email ingestion, file upload and integrations.
- Prompt assembly: system prompt, templates and conversation history handling.
- Data: retrieval sources, permissions and what the model can see.
- Tools and actions: every function, plugin or API the model can call, and with which credentials.
- Output handling: where responses go (rendered HTML, emails, other systems, code execution).
- Users and roles: who uses it, and with what permissions.
Also agree with stakeholders what harm means for this application: data exposure, unauthorised actions, fraud, harmful content, reputational damage, cost abuse.
Step 2: threat-model with a catalogue
Use a structured catalogue so coverage is systematic rather than ad hoc:
| Category | Example tests |
|---|---|
| Prompt injection (direct) | System prompt extraction, instruction override, role-play jailbreaks |
| Prompt injection (indirect) | Instructions hidden in retrieved documents, emails, web pages, file uploads |
| Sensitive information disclosure | Cross-user and cross-tenant retrieval, secrets in prompts, PII leakage |
| Insecure output handling | XSS through rendered output, injection into downstream queries or commands |
| Excessive agency / tool abuse | Unauthorised tool calls, parameter manipulation, privilege escalation |
| Denial of service and cost | Token flooding, recursive tasks, expensive tool loops |
| Policy and harm | Prohibited content, false commitments, unsafe advice in the domain |
MITRE ATLAS provides a complementary, ATT&CK-style view of adversary tactics and techniques against AI systems.
Step 3: attack, manually and automatically
Manual testing finds the creative, context-specific attacks: persuading a support bot to promise a refund, chaining a retrieval leak with a tool call, or exploiting a business rule.
Automated testing runs large numbers of variations and repeats attacks to measure success rates. Open-source tools have matured this year:
- PyRIT (Microsoft) orchestrates multi-turn attacks and scoring.
- garak probes models for a wide range of known weaknesses.
- promptfoo runs evaluation and red-team suites from configuration, suitable for CI.
# promptfoo-style red-team configuration (illustrative)
targets:
- id: https
config:
url: https://staging.example.com/api/assistant
headers: { Authorization: "Bearer {{env.TEST_USER_TOKEN}}" }
body: { message: "{{prompt}}" }
redteam:
purpose: "Customer support assistant for Example Bank retail customers"
plugins:
- pii # attempts to extract other customers' data
- excessive-agency # attempts to trigger unauthorised actions
- hijacking # off-purpose use
- prompt-extraction
strategies:
- jailbreak
- prompt-injection
numTests: 25Always test in a staging environment with realistic, non-production data and test accounts at every permission level.
Step 4: measure, don't just demonstrate
LLM behaviour is probabilistic. A single successful jailbreak says little about risk; a success rate says a lot.
- Run each significant attack multiple times (for example, 20 to 50 trials).
- Record the success rate, the impact if successful, and the preconditions (authenticated user, specific document present).
- Rate severity on impact and likelihood, as you would any other finding.
quadrantChart title Prioritising LLM findings x-axis Low success rate --> High success rate y-axis Low impact --> High impact quadrant-1 Fix now quadrant-2 Fix architecturally quadrant-3 Monitor quadrant-4 Mitigate and monitor Cross-tenant retrieval: [0.7, 0.95] Refund via injection: [0.35, 0.8] System prompt leak: [0.85, 0.3] Off-topic chat: [0.9, 0.1]
Step 5: fix, then regress
- Prefer architectural fixes: permission filtering, removing tool privileges, human approval, output encoding. They are more robust than prompt changes.
- Treat prompt and filter changes as mitigations, and measure how much they reduce success rates.
- Turn every finding into a regression test that runs in CI whenever prompts, models, tools or retrieval change.
- Retest after model upgrades. Provider updates can reopen issues.
Reporting
Good LLM red-team reports include:
- scope, test accounts and environment,
- methodology and catalogue coverage,
- findings with reproduction steps, success rates, impact and evidence,
- architectural recommendations, not just prompt tweaks, and
- the regression suite handed over to the development team.
The bottom line
Red-teaming an LLM application combines familiar application security testing with new, probabilistic and language-driven techniques. A structured methodology (scope the whole system, work from a catalogue, automate variations, measure success rates and regress every finding) turns it from an ad hoc jailbreak session into assurance you can rely on.
Our application security testing and red teaming services include this methodology for LLM applications and agents.