The training bill gets the headlines, but for almost every company running AI in production the recurring pain is inference cost — the money spent every single time a model answers a request. Training is a capital event you can plan and amortize. Inference is an operating expense that scales with traffic, and it grows quietly with every new feature, every longer prompt, and every user you onboard. In 2026 the inference line has become, for many teams, the fastest-growing item on the cloud bill and the hardest to reason about.
Treating that bill as a fixed cost of doing AI is a mistake. The gap between a naive and a well-engineered serving stack is not a few percent; it is a multiple. The levers are well understood and mostly vendor-neutral: how you batch, cache, quantize, route, and schedule requests, and how honestly you attribute the resulting cost. This post is a practical FinOps playbook for the inference layer, aimed at teams who have shipped something that works and now need it to be affordable at scale — a situation TuniCyberLabs sees constantly with EU clients whose usage has outgrown their first architecture.
Why Inference, Not Training, Is the Cost Center
Most organizations do not train foundation models. They consume them, through an API or by serving open-weight models themselves. The cost that compounds is therefore per-request, incurred forever, and coupled directly to product success — the more your feature is used, the more it costs, which is the opposite of most software. The FinOps Foundation framework offers the right mental model here: inform, optimize, operate. You cannot skip to optimization; you have to make the spend visible first, then reduce it, then keep it reduced as the system evolves.
Where the Money Actually Goes
Before optimizing anything, understand the shape of the bill. Inference cost is not one number; it is several distinct leaks that each need a different fix.
- ▸Tokens, not requests: cost scales with tokens processed. A bloated system prompt or an oversized retrieved context is paid on every call, forever.
- ▸Idle accelerators: a GPU held at low utilization bills the same as a busy one. Poor batching literally burns money on air.
- ▸Output length: generation is autoregressive and output tokens are the expensive ones. Unbounded responses are unbounded cost.
- ▸Retries and duplicates: repeated or near-identical queries recomputed from scratch, often invisibly, under a retry policy nobody reviewed.
- ▸Overprovisioned models: using a frontier model for a task a small model handles well is the single most common form of waste, and the easiest to fix.
The Serving-Layer Levers
For self-hosted models, the serving engine is where the largest structural wins live. These are engineering choices, not procurement choices.
- ▸Continuous batching: modern serving engines interleave requests token by token to keep the accelerator saturated. It is the biggest single throughput win for self-hosted inference.
- ▸KV-cache management: reuse the attention key-value cache across shared prefixes such as system prompts and few-shot examples, so you do not recompute them per request. Paged cache schemes make this practical at scale.
- ▸Quantization: serving weights at lower precision cuts memory and often raises throughput, at a modest and measurable quality cost. Always validate against your evaluation set — never assume the loss is acceptable.
- ▸Speculative decoding: a small draft model proposes tokens that a larger model verifies, cutting latency for the same output distribution.
- ▸Prompt discipline: trim system prompts, cap output length, and compress or summarize retrieved context. The cheapest optimization is the token you never send in the first place.
Routing, Cascades, and Semantic Caching
The next tier of savings comes from not running the big model at all when you do not have to. This is where architecture beats hardware.
- ▸Model routing: send each request to the smallest model that can handle it, classifying by difficulty and escalating only when needed.
- ▸Cascades: try a cheap model first, check its confidence, and fall back to a stronger model only on the low-confidence tail — so you pay frontier prices only for genuinely hard requests.
- ▸Semantic caching: cache answers keyed by embedding similarity rather than exact string match, so paraphrased repeat questions hit the cache.
- ▸Exact caching for determinism: for identical prompts, a plain cache is simply free money left on the table until you claim it.
- ▸The trade-off: routing and caching add complexity and can serve stale or subtly wrong answers. Gate them with confidence thresholds and invalidate caches when the underlying source changes.
Capacity, Hardware, and the Buy-vs-Rent Question
Where and on what you run inference is a cost decision as much as a technical one, and it is where the regional angle bites hardest.
- ▸Autoscale to demand: scale replicas to traffic and scale spiky internal tools to zero. Idle accelerators are the biggest silent cost in most stacks.
- ▸Spot and preemptible capacity: for batch and non-latency-critical inference, interruptible instances cut cost sharply for a little orchestration work.
- ▸Right-size the accelerator: match the model and its quantization to the cheapest device that still meets your latency SLO, not the largest device available.
- ▸Self-host versus managed API: managed APIs win on time-to-market and elasticity; self-hosting open-weight models wins on unit cost at steady high volume and on data residency. The crossover point is a real calculation, not a matter of taste.
- ▸Attribute cost as a first-class signal: tag tokens and dollars per feature and per customer using OpenTelemetry-based instrumentation, because you cannot optimize what you cannot attribute.
For teams in Tunisia and North Africa, inference economics cut two ways. Nearshore MLOps talent building for EU clients can turn disciplined serving into a genuine cost advantage, delivering the same SLOs at lower total cost than higher-wage regions. At the same time, local power and accelerator scarcity, together with the need for EU data residency, push many teams toward self-hosting compact open-weight models in-region rather than routing every token to a distant API — a choice that satisfies GDPR residency expectations and controls unit cost at once. For startups operating under Tunisia's Startup Act, where runway is scarce, an inference bill that scales linearly with users can decide whether the unit economics ever close.
A 30/60/90 Cost-Control Roadmap
1. Days zero to thirty: instrument. Attribute token count and cost per feature, per model, and per customer, and set p95 and p99 latency SLOs so later optimizations cannot silently break the user experience. 2. Days zero to thirty: cap output length and trim system prompts across every call path — the fastest no-regret savings available. 3. Days thirty to sixty: add exact and semantic caching for repeat and paraphrased queries, and track the hit rate as a real metric. 4. Days thirty to sixty: introduce model routing so easy requests hit small models, validating quality against your evaluation set at every step. 5. Days sixty to ninety: for self-hosted paths, enable continuous batching and KV-cache reuse, then evaluate quantization behind explicit quality gates. 6. Days sixty to ninety: move batch and non-interactive workloads to spot capacity and autoscale interactive services to demand. 7. Ongoing: review a cost-per-outcome dashboard every month in a FinOps cadence, alongside reliability and quality, so savings do not decay.
What to Do This Week
- ▸Put a hard maximum on output tokens for every endpoint today; it is a one-line change with immediate, permanent savings.
- ▸Add cost and token attribution to your traces so next week's decisions are driven by data rather than intuition.
- ▸Identify your single most expensive endpoint and ask honestly whether a smaller model or a cache could serve most of its traffic.
The direction of 2026 is that AI inference is being absorbed into the same FinOps discipline that tamed the cloud bill a decade ago: measured per unit of business value, attributed to teams, and continuously optimized rather than treated as a mysterious fixed cost. Expect cost-per-outcome to sit next to latency and quality as a governed service-level objective, expect routing and caching to become default architecture rather than advanced tricks, and expect efficiency — smaller models, better scheduling, in-region hardware — to become a competitive edge. The teams that win will be the ones who made inference boring, observable, and cheap while everyone else was still paying frontier prices for every hello.