Most teams test their LLM features by eyeballing a few prompts before launch, then hoping nothing changes. Prompt injection does not wait for launch, and models drift with every provider update. A test suite that runs on every pull request turns an unbounded risk into a bounded, observable one.
What does a prompt injection test suite actually test?
A prompt injection test suite verifies that untrusted text cannot override your system instructions, exfiltrate secrets, or trigger unauthorized tool calls. It treats every input and every retrieved document as hostile, then asserts that the application refuses, sanitizes, or contains the attempt rather than complying with it.
- ▸It tests the application, not the raw model: your system prompt, guardrails, retrieval layer, and tool-calling policy working together.
- ▸It maps to OWASP LLM Top 10 (LLM01: Prompt Injection) and the risk categories in the NIST AI RMF, so findings map to a recognized taxonomy.
- ▸It separates direct injection (the user types the attack) from indirect injection (the attack hides in a retrieved page, PDF, email, or database row).
For the conceptual grounding behind why this is hard, see Prompt Injection Defense in Depth: Assume the Text Is Hostile and the plain-language primer Prompt Injection: The Top New Vulnerability in AI-Powered Apps.
Which categories of injection tests belong in the suite?
Cover at least six categories: instruction override, system-prompt leakage, data exfiltration, indirect or data-borne injection, tool and function-call abuse, and obfuscation or encoding bypass. Each category needs several payloads and one clear expected outcome, so a single model quirk cannot silently pass an entire class of attack.
- ▸Instruction override: "ignore previous instructions", role-play jailbreaks, and persona attacks that try to unseat the system prompt.
- ▸System-prompt leakage: attempts to make the model print its own hidden system or developer instructions.
- ▸Data exfiltration: coaxing the model to emit secrets, another user's data, or a planted canary string.
- ▸Indirect injection: malicious instructions hidden in retrieved documents, HTML comments, image alt text, or tool output.
- ▸Tool and function-call abuse: forcing an agent to invoke a dangerous tool (send_email, delete, transfer) it should refuse.
- ▸Obfuscation: base64, homoglyphs, translation, markdown and HTML smuggling, and split-token payloads that evade naive filters.
How do you assert on a non-deterministic model?
You assert on behavior, not exact strings. Combine deterministic checks (regex, JSON-schema validation, canary-token absence, tool-call allowlists) with an LLM-as-judge for fuzzy refusals, and set temperature to zero plus a pinned model version so runs are reproducible enough to gate a build.
- ▸Deterministic assertions: the output must not contain a seeded canary secret, must not call a denied tool, and must return valid JSON against a schema.
- ▸Refusal classification: an LLM-as-judge (a cheaper model with a strict rubric) scores whether the output complied or refused; promptfoo, DeepEval, and Giskard all ship this pattern.
- ▸Reduce variance: temperature 0, a fixed seed where the provider supports it, a pinned dated model snapshot, and three to five repetitions with a pass threshold instead of a single lucky sample.
Assertions are only trustworthy if you can see what the model actually did, so pair them with tracing as described in LLM Observability: You Cannot Debug What You Did Not Trace.
What fixtures and payloads should you seed?
Seed a versioned corpus of attack payloads and benign controls. Each fixture pairs an input, the context it arrives in, and the expected verdict: refuse, sanitize, or comply safely. Include benign prompts that merely look adversarial so you also catch over-refusal, which is a real regression in its own right.
- ▸Store payloads as data (YAML or JSON), not hard-coded strings, so security reviewers can add cases without touching test code.
- ▸Pull from public corpora such as garak (NVIDIA) probes and PromptInject- or HackAPrompt-style datasets, then add payloads specific to your own domain and tools.
- ▸Embed canary tokens (unique random strings) in system prompts and retrieved documents; any appearance in output is an unambiguous exfiltration failure.
- ▸Keep a benign control set so you can measure the false-positive rate and prove the model still helps real users.
How do you test indirect injection and tool abuse?
Stand up the real retrieval and tool layer with mocked backends, then plant malicious instructions inside the documents and tool outputs the agent will read. Assert that planted instructions never change the tool-call plan and that denied tools are never invoked, using a recorded trace of every call the agent attempts.
- ▸For RAG, inject payloads into indexed documents, HTML comments, and metadata, then confirm the answer ignores the instructions and cites only legitimate content.
- ▸For agents, wrap each tool in a harness that records its arguments, then assert against a tool allowlist and argument constraints (no external recipients, no destructive verbs).
- ▸Apply least privilege so even a successful injection has a small blast radius; the scoping patterns are covered in Least Privilege for AI Agents: Scoping Tools, Tokens, and Blast Radius.
- ▸Simulate multi-step attacks where the output of an early tool carries the payload that fires on a later step.
How do you wire the suite into CI without flaky builds?
Run the suite as a dedicated CI job on pull requests and on a nightly schedule. Cap cost with a small but representative subset per PR, run the full corpus nightly, cache responses where you can, and gate merges on a scored threshold rather than an all-or-nothing pass or fail.
- ▸Drive it from pytest or the native runner in promptfoo or DeepEval, invoked by GitHub Actions, GitLab CI, or your pipeline of choice.
- ▸Use two tiers: a fast smoke set of a few dozen payloads blocks PRs, while the full set of hundreds runs nightly and on release branches.
- ▸Store provider keys as CI secrets and set a spend cap and timeout so a hung model never stalls the pipeline.
- ▸Publish results as a build artifact and trend the attack success rate; a regression in that number fails the build.
- ▸Re-run the whole suite whenever you bump the model version or edit the system prompt, since those are the changes most likely to reopen a closed hole. This is the automated counterpart to the manual work in AI Red-Teaming: Stress-Testing the Models and Agents You Ship.
What should you measure and gate on?
Gate on attack success rate (trending toward zero), over-refusal rate on the benign control set, and canary-exfiltration count (which must be zero). Track each metric per category so a rise in one attack class is visible immediately, and treat any new canary leak or denied-tool call as a hard build failure.
- ▸Hard-fail signals: a canary token appears in output, a denied tool is called, or a secret pattern leaks.
- ▸Trend signals: category-level success rate, over-refusal rate, and median cost and latency per run.
- ▸Tie every failure to a ticket and a concrete fix (stronger system prompt, input filter, output filter, or tool policy), then add the exact bypass as a permanent regression test so it can only be discovered once.
How TuniCyberLabs helps
We build these suites and wire them into your pipeline: a versioned payload corpus mapped to OWASP LLM01, deterministic plus judge-based assertions, canary-based exfiltration checks, and CI gates that fail on regression, integrated with your existing evals and observability so security keeps pace with every model change.
Talk to our AI security engineers about a prompt-injection test suite for your stack at /services.