Why does inference, not training, dominate your AI bill?
For companies that consume foundation models rather than pretrain them, inference is the majority of GPU spend, public reporting commonly puts it around 80% or more of production AI cost. Training is a one-time capital event; inference is a per-request tax you pay for the life of the feature. Optimize the serving path first.
- ▸Training is bursty, inference is perpetual. A fine-tune runs for hours; a chat feature serves tokens every second of every day. Small per-token inefficiencies compound into the largest line on the invoice.
- ▸Cost scales with usage, not headcount. As adoption grows, spend grows super-linearly if prompts get longer or you add retrieval context. Governance on prompt and context size is a cost control, not just a quality one.
- ▸The unit that matters is the token, split two ways. Input (prompt and prefill) tokens and output (decode) tokens have different cost profiles; output tokens are usually far more expensive because decoding is sequential and memory-bound.
For the architecture-level view of what a model actually costs to run, see The Real Cost of Running an LLM in Production in 2026.
How do you measure inference cost per token honestly?
Compute cost per 1,000 tokens by dividing the fully loaded hourly cost of the serving fleet by the tokens it actually produces per hour at your real traffic shape. The denominator, effective throughput under concurrency, is where teams fool themselves; a GPU benchmarked at batch size 1 costs many times more per token than the same GPU saturated.
- ▸Track the three metrics that drive cost: TTFT (time to first token), TPOT (time per output token), and end-to-end throughput in tokens per second. Cost and latency trade against each other through batch size.
- ▸Use FOCUS-aligned billing exports. The FinOps Foundation's FOCUS spec normalizes cloud cost data so GPU line items are comparable across AWS, Azure, and GCP. Tag inference workloads so per-feature unit economics are visible.
- ▸Measure at your percentiles, not the vendor's. A throughput number at batch 256 is meaningless if your p95 concurrency is 8. Load-test with your real prompt and response length distribution.
Unit-economics discipline carries over directly from Kubernetes FinOps: From Cluster Bill to Unit Economics.
What is continuous batching, and why does vLLM cut cost most?
Continuous (in-flight) batching lets the server admit and evict requests every decoding step instead of waiting for a fixed batch to finish, keeping the GPU saturated. Combined with PagedAttention KV-cache management, it is why vLLM and similar engines routinely deliver several times the throughput of naive Hugging Face generation on the same hardware.
- ▸vLLM popularized PagedAttention, which stores the KV cache in non-contiguous blocks and cuts memory fragmentation, so more requests fit in VRAM concurrently.
- ▸Alternatives worth benchmarking: NVIDIA TensorRT-LLM (in-flight batching, fused kernels), SGLang (RadixAttention for aggressive prefix reuse), and Hugging Face TGI. The right engine depends on model, hardware, and prompt shape.
- ▸Throughput is the lever. Higher sustained batch size means more tokens per GPU-hour, which directly lowers cost per token, until latency SLOs push back. Tune batch size to your p95 TTFT budget, not to a leaderboard.
How much can caching and routing save before you touch hardware?
Often more than the hardware changes do. Prefix and KV caching avoid recomputing shared prompt prefixes; semantic caching serves repeat questions with no generation at all; and model routing sends easy requests to small cheap models and hard ones to large models. Stack these before buying a single GPU.
- ▸Prefix caching (native in vLLM and SGLang's RadixAttention) reuses the KV cache for shared system prompts and few-shot examples, high leverage when every request carries the same long instructions.
- ▸Semantic caching (the GPTCache pattern) returns a stored answer when a new query is semantically near a previous one. Set similarity thresholds conservatively to avoid wrong-answer cache hits.
- ▸Model routing and cascades (via LiteLLM or a custom router) try a small model first and escalate only on low confidence. Most production traffic is easy; do not pay flagship prices for "reset my password."
- ▸Shrink the prompt. Trim retrieval context, dedupe boilerplate, and cap output length. Fewer tokens is the cheapest optimization of all.
More on adding capability without runaway spend: How to Add AI Features to Your Product Without Overspending.
Does quantization lower cost without wrecking quality?
Usually yes, within limits. Quantizing weights and activations to FP8 or INT8 roughly halves memory and raises throughput with minimal quality loss on most tasks; INT4 (AWQ, GPTQ) goes further but needs task-specific evaluation. Always gate a quantized model behind your eval suite before it serves traffic.
- ▸FP8 on Hopper-class GPUs (H100, H200) and INT8 are the safe defaults, large memory and throughput gains for small, measurable degradation.
- ▸INT4 weight-only (AWQ, GPTQ) cuts VRAM enough to fit bigger models on smaller cards, but quality varies by task; reasoning and code are more sensitive than summarization.
- ▸Speculative decoding (a small draft model proposes tokens the big model verifies) can raise throughput with no quality change, because the target model still validates every token.
- ▸Never ship on vibes. Keep a golden eval set and regression-test every quantization or engine change, exactly as you would code.
When does self-hosting beat per-token API pricing?
Self-hosting wins when utilization is high and sustained. The break-even is blunt: a reserved or owned H100-class GPU costs a roughly fixed amount per hour whether idle or busy, so it only beats per-token APIs once you keep it busy enough that its cost per token drops below the API price. Bursty, low-volume workloads favor APIs.
- ▸Do the arithmetic per token, not per month. Take the fully loaded GPU-hour cost (hardware or reserved instance, power, ops, a utilization haircut), divide by realistic tokens per hour at your batch size, and compare to the managed per-token rate.
- ▸Utilization is everything. A GPU sitting at 15% utilization is more expensive per token than most APIs. Scale-to-zero and consolidation are what make self-hosting pay.
- ▸Count the hidden costs: MLOps engineering, on-call, model updates, security patching, and capacity headroom. Managed APIs bundle these into the token price.
- ▸Sovereignty can change the math. EU data-residency requirements may justify self-hosting even below break-even. When the workload is steady and bare metal wins outright, see Cloud Repatriation: When Bare Metal Wins the Math.
For a complementary cost-control checklist across the stack, see AI Inference Cost Control: A FinOps Playbook for 2026.
How do you keep GPUs busy without paying for idle?
Match capacity to demand with autoscaling, scale-to-zero, and cheaper capacity classes. Idle accelerators are the number-one source of waste. Drive autoscaling from queue depth, consolidate low-traffic models, and buy spot or reserved capacity deliberately rather than running everything on-demand at peak size.
- ▸KEDA scales replicas on queue depth or request concurrency; Karpenter provisions and consolidates GPU nodes and reclaims idle ones.
- ▸Scale-to-zero for spiky endpoints. Accept a cold-start penalty on rarely used models rather than paying around the clock. Keep hot paths warm.
- ▸Right-size the accelerator. Not everything needs an H100, L4, L40S, or A10G may serve small models at a fraction of the cost. Match the card to the model, not to habit.
- ▸Blend purchasing: reserved or committed capacity for the steady baseline, spot for interruptible batch, on-demand only for spikes.
How TuniCyberLabs helps
We engineer inference platforms that hit latency SLOs at a defensible cost per token: benchmarking vLLM, TensorRT-LLM, and SGLang on your workload; building routing, caching, and quantization pipelines gated by eval suites; and modeling the self-host break-even honestly, including EU data-residency constraints. We instrument unit economics so finance and engineering read the same numbers.
Want a cost-per-token audit of your AI stack? Talk to our engineers.