A conventional penetration test ends where the interesting part of a modern AI system begins. The tester reaches the endpoint, confirms authentication holds, notes that the transport is encrypted, and files a report. None of that touches the behavior that actually decides whether a large language model or an autonomous agent is safe to put in front of customers: whether it can be argued out of its own instructions, tricked into calling a tool it should never call, or coaxed into reproducing the data it was trained on. AI red-teaming is the discipline that closes this gap, treating the model and the agent scaffolding around it as an adversarial attack surface in their own right rather than a black box sitting safely behind a firewall.
In 2026 this has moved from research curiosity to a board-level expectation. The EU AI Act's obligations for high-risk and general-purpose systems assume that providers can demonstrate adversarial testing, not just accuracy metrics. The NIST AI Risk Management Framework builds its Measure function around exactly this kind of evidence. And practitioners now have shared vocabularies to work from: the OWASP Top 10 for LLM Applications, the OWASP Machine Learning Security Top 10, and MITRE ATLAS, the adversarial-threat knowledge base that does for AI systems what earlier technique catalogs did for enterprise intrusions. AI red-teaming is where those frameworks stop being documents and become a repeatable test plan.
Why classic pentesting stops at the model boundary
A traditional assessment optimizes for a clean finding: an unauthenticated endpoint, an injectable parameter, a missing patch. Those still matter, but they say nothing about a system whose most dangerous inputs are grammatically valid English sentences.
- ▸The vulnerability is probabilistic: A model does not fail deterministically. The same jailbreak may succeed one time in five, which means a single test run proves very little and coverage has to be measured statistically.
- ▸The input space is unbounded: You cannot enumerate every prompt. Effective testing works from a taxonomy of attack classes rather than a fixed payload list.
- ▸The blast radius is the tooling: When a model can call functions, browse, or write to a database, a successful manipulation is no longer a bad sentence. It is an unauthorized action with real side effects.
The AI attack surface, mapped
MITRE ATLAS organizes the landscape into tactics that will feel familiar to anyone who has read an intrusion matrix, adapted for machine learning. A working red team exercises each of them deliberately.
- ▸Prompt injection and jailbreaks: Direct and indirect manipulation that overrides system instructions, including payloads hidden in documents, web pages, or retrieved context that the user never sees.
- ▸Training-data extraction: Coaxing a model to emit memorized secrets or personal data, a direct GDPR exposure when the corpus included regulated information.
- ▸Membership inference and model inversion: Determining whether a specific record was in the training set, or reconstructing sensitive attributes from model outputs.
- ▸Data poisoning: Corrupting fine-tuning or retrieval sources so the model behaves badly on attacker-chosen triggers while looking healthy on standard evaluations.
- ▸Model extraction: Cloning a proprietary model's behavior through systematic querying, turning your API into someone else's training set.
Mapping findings to ATLAS techniques gives the same benefit that technique-mapping brought to network defense: a coverage matrix instead of an anecdote, and a shared language between the people breaking the system and the people expected to fix it.
Building an AI red-team program
A one-off jailbreak demo impresses a demo audience and changes nothing. A program produces evidence that improves release after release.
1. Define the trust boundary explicitly: list every input the model consumes, including retrieved documents and tool outputs, and every action it can take. 2. Assemble an attack taxonomy from the OWASP LLM Top 10 and MITRE ATLAS so testers work from classes, not a stale payload file. 3. Automate the repeatable part with adversarial harnesses that fuzz prompts, rerun known jailbreak families, and measure success rates across many trials. 4. Keep humans on the creative edge, where the novel multi-turn manipulations and social-context attacks live that automation will not discover on its own. 5. Score every finding by success probability and by the real-world impact of the action it unlocks, not by novelty. 6. Feed results into guardrails and evaluations, then rerun the same suite to confirm the mitigation actually moved the success rate. 7. Gate releases on the evidence, treating a regression in attack-success rate the way you would treat a failing security test in a build pipeline.
Red-teaming agents, not just prompts
The fastest-growing risk in 2026 is not the chatbot that says something embarrassing. It is the autonomous agent that acts. Once a model can invoke tools, query systems, and chain steps, prompt injection becomes remote code execution by another name, and the interesting question is what an attacker can make the agent do.
- ▸Tool-abuse chains: An injected instruction inside a retrieved email or web page that tells the agent to exfiltrate data through a legitimate tool it already holds.
- ▸Confused-deputy escalation: The agent using its own elevated permissions on behalf of a low-privilege attacker, the classic authorization gap reborn in an AI wrapper.
- ▸Excessive agency: Autonomy and permissions broader than the task requires, so a single manipulation reaches far beyond its intended blast radius.
The durable defenses here are old ideas applied to new actors. Least-privilege scoping of every tool an agent can reach, human-in-the-loop confirmation for irreversible actions, and Zero Trust principles from NIST 800-207 extended so that the agent is a workload with its own scoped identity rather than a trusted insider. Runtime controls help too: sandboxed execution, egress allow-lists, and eBPF-based monitoring that flags when an agent process makes a network call no one authorized.
Scoring findings when CVSS was not built for this
Security teams reach instinctively for CVSS, and it does not fit cleanly. CVSS assumes a deterministic vulnerability with a stable exploitability, while a jailbreak has a success probability that shifts with every model update. A finding that a guardrail can be bypassed one attempt in three is real and serious, but the vector string was never designed to express it.
The pragmatic answer is a two-axis score that speaks to engineers and executives alike. One axis is attack-success rate under a defined number of trials, tracked over time so you can prove a mitigation worked. The other is impact, and impact should be defined by the action the exploit unlocks, reading the data classification and the reversibility of any tool call, rather than by the cleverness of the prompt. That framing keeps the conversation on business risk and gives you a metric you can actually trend across releases.
What to do Monday
You do not need a large budget to start, only discipline and a place to begin.
- ▸Inventory your AI systems, including the shadow ones: every model endpoint, every agent, every retrieval pipeline touching sensitive data.
- ▸Run a known jailbreak suite against your highest-risk system this week and record the success rate as a baseline, not a pass or fail.
- ▸Draw the tool-permission map for any agent in production and remove every capability the task does not strictly require.
- ▸Add adversarial cases to your evaluation set so regressions are caught in the pipeline, not by a customer.
- ▸Assign an owner, because AI safety with no name attached is a document, not a control.
The nearshore assurance opportunity for Tunisia and North Africa
The EU AI Act is manufacturing demand for independent AI assurance faster than Europe can staff it, and that demand does not have to be met exclusively from within the bloc. Tunisia has an unusually deep offensive-security talent pool, sustained by a strong university pipeline and an active capture-the-flag culture, and AI red-teaming is a natural extension of skills that already exist there. For European clients, a Tunisian partner offers meaningful cost advantage, overlapping time zones, and French, Arabic, and English coverage that suits both EU and regional MENA engagements.
The catch is that this work is data-sensitive by nature. Red-teaming a model trained on European personal data pulls GDPR, and increasingly the AI Act, directly into the engagement. The firms that will win this work are the ones that treat data residency as a design constraint: testing against sanitized or synthetic corpora where possible, keeping regulated data inside EU-hosted environments, and documenting the whole arrangement under standard contractual clauses and a proper data-processing agreement. Handled that way, AI assurance becomes one of the clearest nearshoring opportunities the region has, high-skill work that plays to local strengths and that European buyers genuinely cannot get enough of.
The trajectory is clear. As regulators harden their expectations and agents take on more consequential actions, adversarial testing of AI will follow the same path application security took a decade ago, from an optional audit to a gate in every pipeline. The teams that build the muscle now, a living attack taxonomy, automated success-rate tracking, and least-privilege agents by default, will ship AI that survives contact with real adversaries. Those who keep treating the model as a black box behind a firewall will keep discovering, in production and in the headlines, that the most dangerous input to their system was a perfectly ordinary sentence.