When does self-hosting Gemma or Qwen beat a managed API?
Self-host when data residency, cost at scale, or model control outweighs operational convenience. If EU-origin data cannot legally leave your infrastructure, or you run high, steady inference volume, an on-prem or EU-region GPU deployment of Gemma 3 or Qwen2.5 usually wins. For spiky, low-volume workloads, a managed API is cheaper and simpler.
- ▸Residency and sovereignty: GDPR, NIS2, and sector rules for health, finance, and public sector can require that personal or classified data never leaves a defined boundary. Self-hosting removes the third-party processor entirely.
- ▸Cost at scale: a fixed GPU cost amortizes below per-token pricing once utilization is high and steady.
- ▸Control: pinned model versions, no silent deprecations, custom fine-tunes, and full latency control.
For the wider architecture, read Sovereign Cloud and EU Data Residency in 2026: An Engineering Playbook.
Which models and licences are safe to self-host for commercial EU use?
Gemma 3 (1B, 4B, 12B, 27B) and Qwen2.5 (0.5B to 72B) are both strong, multilingual, and self-hostable, but read the licence before you ship. Qwen2.5 is Apache-2.0 for most sizes, while Gemma ships under Google's Gemma Terms, which permit commercial use but impose use restrictions. Always verify the specific release's licence file.
- ▸Qwen2.5 and Qwen3: Apache-2.0 on most variants (verify per size, as some differ), with strong multilingual and coding performance.
- ▸Gemma 3: Google's Gemma Terms of Use allow commercial use with a prohibited-use policy you must pass through to your own users.
- ▸Also viable: Mistral (Apache-2.0 for many models) and Llama 3.x (Llama Community Licence, with a monthly-active-user threshold). Keep a record of the exact licence and version, because auditors and enterprise customers will ask.
How much GPU do you actually need to serve a 7B to 27B model?
A rough rule: in 16-bit precision, budget about 2 GB of VRAM per billion parameters plus headroom for the KV cache. A 7B model needs roughly 16 to 20 GB; a 27B model needs about 55 to 60 GB or multiple GPUs. Quantization cuts this sharply, letting a 7B model run comfortably on a single 24 GB card.
- ▸7B to 9B (Gemma 3 9B, Qwen2.5-7B): one NVIDIA L4 (24 GB) or A10G when quantized; an L40S or A100 40 GB for full precision and high throughput.
- ▸12B to 14B: a single A100 40 GB, or 24 GB with 4-bit quantization.
- ▸27B and above: A100 or H100 80 GB, or tensor-parallel across two 40 GB GPUs.
- ▸KV cache is the hidden cost: long contexts and high concurrency consume VRAM fast, so size for peak concurrent tokens, not just the model weights.
- ▸Throughput, not just fit: two people asking whether a model fits a card usually mean different things. Fitting the weights is the floor; sustaining your target tokens-per-second at your concurrency is the real constraint, and it is what decides how many GPUs you buy.
How does quantization change your GPU sizing and quality?
Quantization stores weights at lower precision (INT8, INT4, FP8) to cut VRAM and boost throughput, usually with minor quality loss at 8-bit and modest, task-dependent loss at 4-bit. It is what lets a 14B model fit on a 24 GB GPU. Always benchmark the quantized model on your own eval before trusting it.
- ▸Formats: GGUF for llama.cpp and Ollama, AWQ and GPTQ for 4-bit GPU-optimized weights that load in vLLM, and FP8 on Hopper and Ada GPUs.
- ▸Sweet spot: 4-bit (a Q4_K_M-class GGUF or AWQ) typically retains most quality for extraction, classification, and chat; test before assuming it holds for reasoning-heavy tasks.
- ▸Throughput bonus: quantized weights free VRAM for the KV cache, so you serve more concurrent requests per GPU.
Why is vLLM the default serving engine, and how do you configure it?
vLLM is the default because PagedAttention and continuous batching give it high throughput and efficient KV-cache use, and it exposes an OpenAI-compatible API so your application code barely changes. It serves Gemma and Qwen out of the box, supports tensor parallelism, and loads AWQ, GPTQ, and FP8 quantized weights.
- ▸Continuous batching: requests join and leave batches dynamically, keeping the GPU busy, which is the main reason vLLM beats naive serving.
- ▸Key knobs: tensor-parallel size across GPUs, maximum model length (which bounds the KV cache), the GPU memory utilization fraction, and the quantization method.
- ▸Alternatives: Ollama or llama.cpp for single-node and desktop simplicity, Hugging Face TGI as another production server, and SGLang for structured or agent workloads. For a scaled service, vLLM behind a load balancer is the common choice.
Before you commit, sanity-check the economics in The Real Cost of Running an LLM in Production in 2026.
How do you prove EU data residency to auditors?
Residency is proven with documented data flows, region-pinned infrastructure, and access controls you can evidence, not a vendor's marketing claim. Deploy in an EU region or on-prem, keep prompts and logs inside that boundary, and produce a data-flow diagram plus records of processing (GDPR Article 30) that show no data leaves it.
- ▸Pin the region: an EU cloud region or an EU or national datacentre, with cross-region replication disabled for inference data.
- ▸Contain the logs: prompts and completions are personal data when they contain personal data, so keep observability, traces, and vector stores in-region too.
- ▸Evidence access: IAM policies, encryption at rest and in transit, and audit logs, mapped to NIS2 and, for finance, DORA.
- ▸Verify against primary sources: EU rules and guidance shift, so confirm current obligations against official EU and national DPA texts, since timelines remain in flux.
For the business case behind all this, see EU Data Residency: Why Where Your Data Lives Matters in 2026.
What does a production on-prem inference stack include?
Beyond the model and vLLM, a production stack needs a gateway, autoscaling, observability, and guardrails. Treat the LLM as one service in a normal platform: an API gateway for auth and rate limiting, GPU-aware scaling, tracing and evals, and input and output filtering. Without these, self-hosting trades an API bill for an outage.
- ▸Gateway and auth: an OpenAI-compatible gateway (LiteLLM or your own) for keys, quotas, and routing across models.
- ▸Scaling: Kubernetes with GPU node pools and KEDA or HPA, or a dedicated GPU host, scaling on queue depth and latency rather than CPU alone.
- ▸Observability: request tracing, token and latency metrics, and continuous evals to catch regressions after every model or config change.
- ▸Guardrails: prompt-injection defenses, output validation, and least privilege on any tools the model can call.
- ▸Capacity planning: model peak concurrent requests and average token length, then provision GPUs with headroom for spikes and rolling upgrades. Batching hides latency at moderate load and collapses it under saturation, so alert on queue depth and time-to-first-token, not just error rate.
Keep sensitive prompts from leaking through logs and tools; see How to Safely Integrate LLMs Into Your Product Without Leaking Data.
How TuniCyberLabs helps
We build EU-resident inference platforms end to end: model and licence selection, GPU sizing, quantization benchmarks, vLLM deployment on your cloud region or on-prem hardware, and the gateway, scaling, observability, and guardrails around it, with the Article 30 records and control mapping your auditors will ask for. Engineering is delivered from our Tunisia team for the EU and North Africa.
Send us your residency constraints and expected token volume, and we will design the GPU footprint and deployment: see our services.