Admission control and image scanning stop known-bad workloads before they run. Runtime threat detection catches what happens after: the exploited dependency, the cryptominer pulled into /tmp, the reverse shell inside a pod that passed every scan. This playbook covers Tetragon - the eBPF-based runtime security tool from the Cilium project - with the policy patterns, enforcement decisions, and operating costs that matter in production.
What is Tetragon and why use it for runtime detection?
Tetragon is an open-source, eBPF-based security observability and enforcement tool from the Cilium project. It runs as a DaemonSet on each Kubernetes node, hooks syscalls and kernel functions in-kernel, filters events before they reach userspace, and can kill an offending process synchronously - detection and response without per-pod agents, sidecars, or image changes.
The architectural difference from older tools matters. Falco, the other widely deployed option, streams syscall events into userspace and evaluates rules there; by default it alerts, and relies on a companion project (Falco Talon) to respond. Tetragon pushes filtering into the kernel with policy selectors, so only events that match a policy cross the kernel boundary at all, and enforcement happens inline - a Sigkill lands before the hooked operation completes. KubeArmor takes a third route, enforcing through Linux Security Modules such as AppArmor and BPF LSM.
We mapped the broader detection landscape in Runtime Security with eBPF: Detecting Kubernetes Threats in 2026. This playbook goes narrower and deeper: how to actually run Tetragon.
How does Tetragon observe syscalls without per-pod agents?
Every container on a node shares that node's kernel, so one eBPF agent per node sees every process exec, file access, and network connection in every pod. Tetragon loads programs at kprobes, tracepoints, and LSM hooks, then enriches each event in-kernel with Kubernetes identity: pod, namespace, container image, and full process ancestry.
Before you write a single policy, Tetragon already emits process lifecycle events (process_exec and process_exit) with the binary path, arguments, and the parent chain back to the container runtime. That alone answers a question most clusters cannot: what actually executed in this pod, launched by what.
Practical requirements:
- ▸A BTF-enabled kernel. Tetragon's programs use CO-RE (compile once, run everywhere), which needs BTF type information. Most distribution kernels from 5.4 onward ship it; verify against your exact node image, not the distro name.
- ▸No workload changes. No sidecars, no LD_PRELOAD, no image rebuilds. Works with containerd and CRI-O.
- ▸A Helm-based DaemonSet install, plus the tetra CLI for inspecting the live event stream.
How do you write a TracingPolicy?
A TracingPolicy is a Kubernetes custom resource that tells Tetragon what to hook, what to parse, whom it applies to, and what to do. Its four parts: a hook point (a kprobe on a kernel function, a tracepoint, or an LSM hook), typed arguments to extract, selectors that scope the policy in-kernel, and actions.
The selectors are where in-kernel filtering happens:
- ▸matchBinaries - apply only to, or exclude, specific executables by path.
- ▸matchArgs - match on parsed arguments: a file path prefix, a destination port, an address.
- ▸matchNamespaces and matchCapabilities - scope by Linux namespace properties or capability sets.
- ▸matchActions - what to do on match: Post an event, Sigkill the process, Override the return value.
Two field-tested habits. First, prefer hooking LSM and security-layer functions (security_file_permission, security_bprm_check) over raw syscall entry points: syscall arguments live in user memory and can be swapped after inspection - a classic TOCTOU gap - while security hooks see the kernel's resolved objects and stay more stable across versions. Second, use the namespaced variant (TracingPolicyNamespaced) when a policy belongs to one team's workloads; cluster-wide policies should be rare and boring. Keep every policy in Git, review it like application code, and load it against a kind or staging cluster before it meets production kernels.
Which detection policies should you deploy first?
Start with five high-signal, low-noise policies. Public incident reporting on Kubernetes intrusions keeps showing the same post-exploitation moves - cryptominers, reverse shells, credential theft - and these five cover most of them:
- ▸Execution from writable paths (/tmp, /dev/shm, /var/tmp). Catches dropped payloads and miners; production images almost never execute from these directories legitimately.
- ▸Interactive shells in production pods. A bash or sh with a TTY inside a workload pod is either an engineer breaking glass or an attacker; both deserve an event.
- ▸Sensitive file reads - service account tokens under /var/run/secrets, /etc/shadow, cloud credential paths. Catches harvesting before lateral movement.
- ▸Unexpected egress from pods that should never dial out, keyed on tcp_connect with destination filters. Catches reverse shells and miner pool traffic.
- ▸Privilege-change syscalls - setuid, capset, unshare in workloads with no business calling them. Catches escape preparation.
Each needs tuning: CI runner images legitimately execute from temp directories, and some init containers legitimately read token paths. Use matchBinaries exclusions rather than deleting the policy. Then validate the honest way - run the technique yourself in an authorized test and confirm the event fires. We described that loop in From Red Team Findings to Detections: Continuous Purple Teaming.
When should Tetragon enforce instead of observe?
Enforce only after a policy has run in observe mode for typically two to four weeks - long enough to cover deploy cycles, cron jobs, and month-start batch behavior - with a false-positive rate of effectively zero. Enforcement is synchronous: Sigkill terminates the process before the operation completes, and a mistargeted policy is a self-inflicted outage.
The mechanisms differ in blast radius:
- ▸Sigkill kills the process. Simple and drastic.
- ▸Override rewrites the hooked function's return value so the call fails with an error - cleaner for the application, but it only works on kernel functions annotated for error injection, and the kernel must be built with that support.
- ▸LSM-hook enforcement denies at the same layer where AppArmor and SELinux act. Cleanest semantics; requires BPF LSM enabled in the kernel boot parameters, which not every node image does.
Good first candidates are narrow and asymmetric: block execution from /tmp in hardened single-purpose workloads, block unshare-based namespace escapes from application pods, block writes to /proc/sys from anything that is not a named admin tool. Keep observe policies broad and enforcement policies surgical. And remember enforcement at runtime pairs with prevention at deploy time - a workload that admission control rejects never needs killing. That layer is covered in Kubernetes Admission Control and Policy-as-Code in 2026.
What does Tetragon cost to run in production?
Budget three costs. Node overhead: published benchmarks and field reports typically land in the low single digits of CPU percent with a modest policy set, but cost scales with hook frequency - hooking every write is a different world from hooking exec. Pipeline cost: unfiltered JSON export from a busy node can reach thousands of events per second. Engineering time for tuning: usually the largest line item.
Filtering is the lever, at two stages:
- ▸In-kernel selectors drop unmatched events before they cost anything meaningful. Tight matchArgs and matchBinaries clauses are performance features, not just precision features.
- ▸Export filters - allow and deny lists on the exporter - trim what becomes JSON. Sampling process_exit and consciously deciding what to keep from kube-system commonly cuts export volume by an order of magnitude or more.
Do the storage math before rollout: at a hedged 500 to 2,000 events per second per busy node and roughly 1 to 2 KB per JSON event, a 50-node cluster can produce hundreds of gigabytes per day uncompressed if unfiltered. Watch Tetragon's Prometheus metrics for ring-buffer drops - silent event loss under burst load is how detections quietly stop working.
How do you ship Tetragon events into your detection pipeline?
Tetragon writes structured JSON to stdout or a file and serves the same stream over gRPC. The standard pattern: export filters trim the stream, a shipper such as Fluent Bit or Vector forwards it to your SIEM or data lake, and detection rules key on process ancestry plus pod identity.
Three rules keep the pipeline maintainable:
- ▸Write detections against policy names, not kernel function names. Kernels rename and inline functions across versions; your policy names are yours.
- ▸Use the ancestry. A curl is boring; a curl whose parent is a Java web server worker is not. The process chain plus pod and image fields carry most of the detection value.
- ▸Map policies to MITRE ATT&CK techniques for the containers matrix, so SOC triage and coverage reviews speak a shared language.
Keep hot retention short and cold retention long: the stream doubles as a forensic timeline, and when a detection fires you will want the surrounding hours. For the investigation that follows, endpoint-grade collection takes over - see Forensic Triage at Scale with Velociraptor and KAPE.
What are the common failure modes?
Four problems recur: kernel drift, where a policy references a function a newer kernel renamed or inlined and silently stops loading; event loss when ring buffers overflow under burst; noisy untuned policies that bury real signal; and scope confusion - Tetragon observes what its policies name, and nothing else.
The fixes are operational: pin and test policies per node-pool kernel version, alert on policy load failures and on the dropped-event counters, and cap the policy set at what your team actually triages. The scope point deserves emphasis. Tetragon is not a full EDR - it does not scan memory, parse application-layer protocols, or detect business-logic abuse. It is a precise, low-overhead tripwire layer for kernel-visible behavior: exec, file, network, privilege. Treat the policy set like a detection codebase, because that is what it is.
How TuniCyberLabs helps
We design and operate runtime detection for Kubernetes fleets across the EU and North Africa: Tetragon and Falco policy engineering, export pipelines into your SIEM, purple-team validation of every detection, and staged enforcement rollouts that do not cause outages. If your clusters have no runtime visibility today - or a noisy deployment nobody trusts - talk to our engineers and we will scope an assessment.