AI Security

AI Security Testing: How to Red-Team an LLM Application

Most organisations now have LLM features in production, and many have never tested them adversarially. This is a practical methodology for red-teaming an LLM application: scoping, threat modelling, test categories, tooling for automation and how to report probabilistic findings.

By EthicalHat Research · · 3 min read
Infographic: A methodology for red-teaming LLM applications

A year ago, AI red teaming was mostly something model developers did. Today, most organisations deploying LLM features (support assistants, internal copilots, document processing and agents) need to test them adversarially too. Guidance has matured: OWASP's Top 10 for LLM Applications is on version 1.1, MITRE ATLAS catalogues adversarial techniques against AI systems, and last month NIST published its Generative AI Profile (AI 600-1) for the AI Risk Management Framework, which includes red-teaming among its recommended practices.

This article sets out the methodology we use to red-team LLM applications.

Step 1: scope the whole application

The model is only one component. Scope should cover:

  • Entry points: chat UI, API, email ingestion, file upload and integrations.
  • Prompt assembly: system prompt, templates and conversation history handling.
  • Data: retrieval sources, permissions and what the model can see.
  • Tools and actions: every function, plugin or API the model can call, and with which credentials.
  • Output handling: where responses go (rendered HTML, emails, other systems, code execution).
  • Users and roles: who uses it, and with what permissions.

Also agree with stakeholders what harm means for this application: data exposure, unauthorised actions, fraud, harmful content, reputational damage, cost abuse.

Step 2: threat-model with a catalogue

Use a structured catalogue so coverage is systematic rather than ad hoc:

CategoryExample tests
Prompt injection (direct)System prompt extraction, instruction override, role-play jailbreaks
Prompt injection (indirect)Instructions hidden in retrieved documents, emails, web pages, file uploads
Sensitive information disclosureCross-user and cross-tenant retrieval, secrets in prompts, PII leakage
Insecure output handlingXSS through rendered output, injection into downstream queries or commands
Excessive agency / tool abuseUnauthorised tool calls, parameter manipulation, privilege escalation
Denial of service and costToken flooding, recursive tasks, expensive tool loops
Policy and harmProhibited content, false commitments, unsafe advice in the domain

MITRE ATLAS provides a complementary, ATT&CK-style view of adversary tactics and techniques against AI systems.

Step 3: attack, manually and automatically

Manual testing finds the creative, context-specific attacks: persuading a support bot to promise a refund, chaining a retrieval leak with a tool call, or exploiting a business rule.

Automated testing runs large numbers of variations and repeats attacks to measure success rates. Open-source tools have matured this year:

  • PyRIT (Microsoft) orchestrates multi-turn attacks and scoring.
  • garak probes models for a wide range of known weaknesses.
  • promptfoo runs evaluation and red-team suites from configuration, suitable for CI.
yaml
# promptfoo-style red-team configuration (illustrative)
targets:
  - id: https
    config:
      url: https://staging.example.com/api/assistant
      headers: { Authorization: "Bearer {{env.TEST_USER_TOKEN}}" }
      body: { message: "{{prompt}}" }
redteam:
  purpose: "Customer support assistant for Example Bank retail customers"
  plugins:
    - pii              # attempts to extract other customers' data
    - excessive-agency # attempts to trigger unauthorised actions
    - hijacking        # off-purpose use
    - prompt-extraction
  strategies:
    - jailbreak
    - prompt-injection
  numTests: 25

Always test in a staging environment with realistic, non-production data and test accounts at every permission level.

Step 4: measure, don't just demonstrate

LLM behaviour is probabilistic. A single successful jailbreak says little about risk; a success rate says a lot.

  • Run each significant attack multiple times (for example, 20 to 50 trials).
  • Record the success rate, the impact if successful, and the preconditions (authenticated user, specific document present).
  • Rate severity on impact and likelihood, as you would any other finding.
quadrantChart
  title Prioritising LLM findings
  x-axis Low success rate --> High success rate
  y-axis Low impact --> High impact
  quadrant-1 Fix now
  quadrant-2 Fix architecturally
  quadrant-3 Monitor
  quadrant-4 Mitigate and monitor
  Cross-tenant retrieval: [0.7, 0.95]
  Refund via injection: [0.35, 0.8]
  System prompt leak: [0.85, 0.3]
  Off-topic chat: [0.9, 0.1]

Step 5: fix, then regress

  • Prefer architectural fixes: permission filtering, removing tool privileges, human approval, output encoding. They are more robust than prompt changes.
  • Treat prompt and filter changes as mitigations, and measure how much they reduce success rates.
  • Turn every finding into a regression test that runs in CI whenever prompts, models, tools or retrieval change.
  • Retest after model upgrades. Provider updates can reopen issues.

Reporting

Good LLM red-team reports include:

  • scope, test accounts and environment,
  • methodology and catalogue coverage,
  • findings with reproduction steps, success rates, impact and evidence,
  • architectural recommendations, not just prompt tweaks, and
  • the regression suite handed over to the development team.

The bottom line

Red-teaming an LLM application combines familiar application security testing with new, probabilistic and language-driven techniques. A structured methodology (scope the whole system, work from a catalogue, automate variations, measure success rates and regress every finding) turns it from an ad hoc jailbreak session into assurance you can rely on.

Our application security testing and red teaming services include this methodology for LLM applications and agents.

EthicalHat Research

The EthicalHat research team: penetration testers, SOC analysts and engineers writing about security, software engineering and AI.

Want an expert opinion on your own systems?

Email info@ethicalhat.com or call +1-844-838-4422. Our security engineers are happy to talk it through.