What is agentic RAG and how is it different from classic RAG?
Agentic RAG replaces the single retrieve-then-generate step with an LLM agent that plans, retrieves iteratively, and verifies before answering. Classic RAG does one similarity search and hopes it grabbed the right chunks. An agentic system decomposes the question, runs multiple targeted retrievals, checks whether it has enough evidence, and retries or escalates when it does not.
The functional differences:
- ▸Classic RAG: one query, one retrieval, one generation. Fast, but brittle on complex or multi-part questions.
- ▸Agentic RAG: a control loop (often ReAct-style) that reasons about what to fetch next, calls retrieval as a tool, and self-corrects. Patterns include Self-RAG, Corrective RAG (CRAG), and query-planning agents.
- ▸Cost profile: agentic RAG spends more tokens and more round-trips per answer, so you apply it where accuracy and traceability justify the expense, not to every chatbot turn.
This is the retrieval side of the broader shift covered in AI Agents in the Enterprise 2026: From Chatbots to Autonomous Workflows.
How does query decomposition improve multi-step retrieval?
Query decomposition splits a complex question into sub-questions the retriever can answer one at a time, then recombines the evidence. A single dense search for a multi-part question retrieves an averaged, muddy context. Decomposition gives each sub-question a clean, targeted retrieval, which sharply raises recall on multi-hop queries.
How it works in practice:
- ▸Planner step: the agent rewrites *compare our GDPR and NIS2 breach-notification deadlines* into two sub-queries, one per regulation, plus a synthesis step.
- ▸Sequential or parallel retrieval: independent sub-questions run in parallel; dependent ones run in sequence, where the answer to step one shapes the query for step two.
- ▸Sub-answer synthesis: the agent composes verified sub-answers into a final response, each carrying its own citations.
- ▸Query rewriting and HyDE: rewriting vague queries, or generating a hypothetical answer to embed, lifts retrieval precision before the agent ever traverses further.
Decomposition is only as good as the underlying retrieval; the engineering that makes it reliable is in RAG in Production: The Retrieval Engineering Nobody Demos.
What retrieval agent patterns work in enterprise knowledge bases?
The durable patterns are routing, tool-based retrieval, and corrective loops orchestrated by a graph-structured controller. Frameworks such as LangGraph and LlamaIndex give you explicit state, so the agent can branch, retry, and stop deterministically instead of looping forever. Bounded, inspectable control flow is what separates a demo from production.
Patterns that hold up:
- ▸Router agents send a query to the right source: vector store, SQL warehouse, a specific document collection, or a live API, based on intent.
- ▸Corrective RAG (CRAG): grade retrieved chunks for relevance; if they are weak, reformulate the query or fall back to web or another index.
- ▸Self-RAG: the model emits reflection tokens deciding whether to retrieve, and critiques its own draft against the evidence.
- ▸Bounded loops: hard caps on retrieval iterations and a token budget per query prevent runaway cost, echoing the scoping discipline in Least Privilege for AI Agents: Scoping Tools, Tokens, and Blast Radius.
How do you enforce access-controlled retrieval?
Filter documents by the user's permissions at query time, before anything reaches the model, so the agent can only retrieve what the caller is already authorised to see. Post-hoc redaction is a leak waiting to happen. Access control belongs in the retrieval query itself, enforced by your authorization system, not by prompt instructions.
Concrete controls:
- ▸Metadata and ACL filtering: tag every chunk with its source document's permissions and pass the user's identity as a hard filter on the vector query, so restricted vectors are never candidates.
- ▸Relationship-based access control: use a Zanzibar-style engine such as OpenFGA, or policy engines like OPA or Cedar, to resolve *can this user read this document* consistently with the rest of your platform.
- ▸Row-level security at the database for any SQL retrieval path, so the agent inherits the caller's grants.
- ▸Never trust the prompt: an instruction like *only show permitted results* is not a control. Treat retrieved and user text as hostile, as argued in Prompt Injection Defense in Depth: Assume the Text Is Hostile.
Get this wrong and RAG becomes a data-exfiltration channel: one broad question can surface HR files, salaries, or another tenant's records.
How do you verify answers and reduce hallucination?
Ground every claim in retrieved evidence, then run an automated groundedness check that rejects or flags any sentence the sources do not support. Verification is a separate step from generation. The most reliable agentic systems refuse to answer, or escalate to a human, when confidence and evidence are low, rather than fabricating.
Anti-hallucination layers that work together:
- ▸Forced citation: require the model to answer only from retrieved passages and attach source IDs to each claim, so an answer with no citation is a red flag.
- ▸Groundedness or faithfulness scoring: use an NLI-style entailment check or a framework such as RAGAS to verify each sentence is entailed by its cited context.
- ▸Corrective retry: if faithfulness is low, the agent retrieves again or narrows scope before responding.
- ▸Abstention: design an explicit *I could not verify this* path; a refused answer is cheaper than a confident wrong one.
- ▸Tracing: capture every retrieval, tool call, and score with OpenTelemetry, LangSmith, or Arize Phoenix. You cannot debug what you did not trace, as covered in LLM Observability: You Cannot Debug What You Did Not Trace.
What does the agentic RAG reference architecture look like?
A production agentic RAG stack has five layers: an orchestrator, an access-control gate, a multi-source retrieval layer, a verification stage, and full tracing. Each layer is independently testable and independently failable, so a weak retrieval does not silently become a confident hallucination. Design for inspection and bounded cost from day one.
The layers:
- ▸Orchestration: LangGraph or LlamaIndex holds explicit state, plans sub-queries, and enforces iteration and token caps.
- ▸Authorization gate: resolves the caller's identity and injects permission filters into every retrieval call.
- ▸Retrieval: hybrid dense-plus-keyword search with a cross-encoder reranker, plus routing to SQL, graph, or API sources as needed.
- ▸Verification: citation enforcement, groundedness scoring, and an abstention path.
- ▸Observability: end-to-end traces, per-query cost, faithfulness metrics, and evaluation gates in CI so regressions are caught before release.
How do you keep agentic RAG cheap and fast enough?
Reserve the full agent loop for hard queries and serve everything else with a cheap single-shot path. A router that sends simple lookups to classic RAG and only escalates complex questions to the agent keeps median latency and cost low while preserving accuracy where it matters. Budget every query and cap iterations.
Cost and latency tactics:
- ▸Tiered routing: classify query complexity first; most turns never need decomposition or multiple retrievals.
- ▸Model tiering: use a small, fast model for planning and grading, and a stronger model only for final synthesis.
- ▸Caching: cache embeddings, sub-query results, and frequent answers to skip repeated retrieval.
- ▸Hard budgets: enforce a maximum iteration count and token ceiling per query so a stuck agent fails fast instead of billing indefinitely.
- ▸Parallelism: run independent sub-queries concurrently to cut wall-clock latency without cutting steps.
How TuniCyberLabs helps
We build access-controlled, verifiable agentic RAG for regulated EU and North African organisations: query-decomposition planners, ACL-aware retrieval wired to OpenFGA, OPA, or your existing IAM, groundedness verification with abstention, and full tracing with cost budgets. You get an architecture that answers hard questions, proves its sources, and never returns data the caller should not see.
Building a knowledge base your team can trust with real permissions and no hallucinations? Book an agentic RAG design session and we will map your sources, access model, and verification gates into a reference architecture you can ship.