The instrumentation tax
Two hundred services. Five languages. An OpenTelemetry SDK in every one of them, each pinned to a different version, each upgrade a ticket that nobody owns. The Go services have tracing because one team cared. The Python ones have half of it. The vendor Java app has none, because you cannot change its code. Every incident review ends with the same line: the trace stopped at the service we never instrumented.
Zero-code tracing promises a way out. Run one agent per node, as a DaemonSet. Get traces and RED metrics (rate, errors, duration) for every service on the node. No code changes, no restarts, no SDK upgrades.
The project making that promise in the open is OpenTelemetry eBPF Instrumentation, or OBI. Grafana donated it from Beyla, its eBPF auto-instrumentation tool, in May 2025. Beyla still exists as Grafana's distribution, built on OBI. OBI's first alpha shipped in November 2025. As I write this, the latest release is v0.13.0, from September 2026, and a stable 1.0 is the SIG's goal for this year. Vendors are already shipping it: Splunk announced a beta of OBI in its own product at KubeCon EU 2026.
So it is about to go stable, and it is worth knowing what it really does before it lands in your cluster. "Zero code" sounds like magic. It is not. It is eBPF hooking a handful of places in your stack, parsing the bytes that pass through them, and, the part most people miss, writing into your traffic to stitch the pieces together.
To show it, I built a small demo: two Go services and Redis, with no tracing code at all. No SDK, no OTel import, no middleware. OBI runs beside them and sends spans to Jaeger.
curl → gateway :8080 ──HTTP──▶ orders :8081 ──RESP──▶ redis :6379
└── /checkout-async → channel → worker goroutine → orders
(OBI runs beside them; spans → Jaeger :16686)One docker compose up, one curl localhost:8080/checkout, and Jaeger shows a single trace that crosses both services and the Redis call. This post walks through how that trace got there, step by step, and then where the approach breaks. There are four interactive visualizers along the way.
jump to: the tax · eBPF in 90s · finding processes · hook points · bytes to spans · propagation · export · where it falls short · my take
1. eBPF in 90 seconds
eBPF lets you run small programs inside the Linux kernel without writing a kernel module. You attach a program to an event: a kernel function being called, a function in a user-space binary being called, a packet arriving. When the event happens, the kernel runs your program inline, on that thread, and then carries on.
Three things make this safe enough to run in production:
- The verifier. Before a program is accepted, the kernel checks every path through it. The program must finish, must not read memory it does not own, and must check pointers before using them. If any check fails, the program is refused. It never runs.
- Maps. A program cannot call
printfor open a socket. The only way data gets in or out is through maps: hash tables, arrays, and ring buffers that both the kernel and user space can see. - The ring buffer to user space. The kernel side does as little as possible: copy a few fields into a fixed-size record and return. An ordinary user-space process, the agent, reads those records and does the expensive work: parsing, joining, building spans, exporting.
Step through one program below, then try the version that forgets a null check.
↳ ebpf in 90 seconds · simplified illustration
SEC("kprobe/tcp_sendmsg")
int on_send(struct pt_regs *ctx) {
struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
if (!e) return 0; // null check: verifier needs it
e->pid = bpf_get_current_pid_tgid() >> 32;
e->bytes = PT_REGS_PARM3(ctx); // size argument
bpf_ringbuf_submit(e, 0);
return 0;
}- 1. loadagent → bpf() syscall
- 2. verifierkernel
- 3. attachkprobe on tcp_sendmsg
- 4. event fireskernel, in the caller’s context
- 5. ring buffershared kernel / user memory
- 6. user spaceagent process
Press step to load the program.
That split matters for everything that follows. The kernel side of OBI is small and fast and sees everything. The user-space side is a normal Go process that you can profile, limit, and crash without taking the node down.
2. Step 1: finding the processes
Before it can hook anything, OBI has to decide what to watch. You tell it with a selector: an open port, an executable name, or Kubernetes metadata such as a namespace or a deployment. The demo uses ports:
obi:
image: otel/ebpf-instrument:v0.13.0
privileged: true # demo only; see the capabilities section
pid: host # see every process on the host
network_mode: host
environment:
OTEL_EBPF_OPEN_PORT: "8080,8081" # instrument gateway and orders
OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost:4318
OTEL_EBPF_BPF_CONTEXT_PROPAGATION: all # off by default; more in section 5Then, for each matching process, it works out three things.
- What language is this? OBI opens
/proc/<pid>/exeand reads the binary's symbols and linked libraries. A Go binary carries its runtime and its symbol table. A Python process linkslibpython. A Java process is a JVM. This is why the agent needsCAP_SYS_PTRACE: it is reading other processes' executables and memory. - Where are the functions? For Go, it finds the addresses of
net/http,crypto/tlsand runtime functions inside the binary. That only works if the symbols are there, which is why the demo's Dockerfile has a warning in it:# NOTE: no -ldflags="-s -w". Keep the symbol table so OBI's Go uprobes can # find net/http, crypto/tls and runtime functions inside the binary. RUN CGO_ENABLED=0 go build -o /out/app ./cmd/${SVC} - Who is this? A PID is not a service. OBI maps PID to cgroup, cgroup to container, container to pod, and pod to a service name and namespace. That name ends up on every span.
Language support today covers Java (JDK 8 and newer), .NET, Go, Python, Ruby, Node.js, C, C++ and Rust. "Support" does not mean the same depth everywhere, though. Go gets the richest treatment, because OBI can hook Go's own functions. For most other languages, OBI relies more on what it sees at the socket and in the TLS library.
3. Step 2: the hook points
Here are the five places in the title. Each one sees a different slice of a request, and OBI combines them.
↳ the five kinds of hook
| Hook | Where | What it sees or does |
|---|---|---|
| kprobes | kernel socket send and receive paths | bytes in and out of every TCP socket, for any language |
| uprobes | Go net/http, gRPC and runtime functions | request boundaries, routes, status, and which goroutine handed work to which |
| uprobes on TLS libraries | OpenSSL, Go crypto/tls, Java TLS | HTTPS plaintext, just before encryption and just after decryption. No keys. |
| socket filters | AF_PACKET raw sockets | HTTP for non-Go languages, and network flows |
| TC, sk_msg, sock_ops | the network path | writes trace context into outgoing traffic |
kprobes are the universal fallback. Every language, in the end, calls into the kernel to send and receive bytes on a socket. A kprobe there sees the bytes, the process, the socket and the timing, whatever the language. What it cannot see is meaning: which bytes belong to which request, or what happens inside the process between receiving a request and sending the next one.
uprobes fill that gap for runtimes OBI knows well. A uprobe on Go's HTTP server handler fires when a request starts and ends, with the route and status right there in memory. A uprobe on the runtime function that starts goroutines tells OBI that goroutine 41 was started by goroutine 17, which was handling request X.
uprobes on TLS libraries are the trick that makes HTTPS visible. Instead of breaking encryption, OBI hooks the function that encrypts: SSL_write in OpenSSL, (*Conn).Write in Go's crypto/tls. At that moment, the argument is plaintext. No keys, no man in the middle.
Socket filters on raw AF_PACKET sockets give OBI a copy of traffic for languages where it has no richer hook, and the data for network flow metrics.
TC, sk_msg and sock_ops programs are the surprise. The first four rows are about watching. This one is about changing: these programs sit on the network path and write trace context into outgoing traffic. OBI does not only observe your packets. It modifies them. Hold that thought for section 5.
The map below follows one outgoing request from gateway to orders down the stack. Switch between HTTP and HTTPS and watch where the plaintext ends. Then switch the propagation mode and watch which hook does the writing.
↳ request hook map · gateway → orders · simplified illustration
- applicationnet/http clientuprobe on net/http (*Transport).roundTrip
- TLS librarycrypto/tlsuprobe on crypto/tls (*Conn).Write
- socketkernelkprobe on tcp_sendmsg
- traffic controlkernel, egressTC egress program
- NICwire—
Step down the stack. Amber marks the hooks that write.
The line that matters is the TLS library. Above it, everything is readable and writable. Below it, the kernel sees only ciphertext. Every design choice in context propagation comes from that one line.
4. Step 3: turning bytes into spans
A kprobe hands the agent bytes, not requests. The agent has to figure out the protocol from the bytes alone. Most protocols give themselves away in the first few bytes: GET / or POST / means HTTP/1.x. A fixed preface means HTTP/2. A * followed by a count means a Redis command array.
The list OBI understands today is long. Client and server side: HTTP and HTTPS, HTTP/2, gRPC, Kafka, NATS, MQTT, Memcached, PostgreSQL, MySQL, MSSQL and Redis. Client side only: MongoDB, Elasticsearch, OpenSearch, AMQP 1.0, DNS, and more.
The demo's Redis client is about thirty lines of hand-written Go, with no Redis library at all. It does not matter. OBI recognizes the protocol on the wire, not the library:
cmd := fmt.Sprintf("*2\r\n$4\r\nINCR\r\n$%d\r\n%s\r\n", len(key), key)
if _, err := conn.Write([]byte(cmd)); err != nil {
return 0, err
}Once the protocol is known, a span is just a few fields pulled out of the request and the response, plus two timestamps: when the request bytes were written and when the response bytes came back.
↳ bytes → span · simplified illustration
GET /orders?fail=1 HTTP/1.1\r\nHost: orders:8081\r\n\r\n
HTTP/1.1 500 Internal Server Error\r\n…
| http.request.method | … |
| url.path | … |
| server.address | … |
| http.response.status_code | … |
| duration | … |
| span.status | … |
Press parse. No SDK, no schema from the app: only bytes and timestamps.
The same event feeds three outputs at once. A span for the trace. RED metrics, because a request count, an error flag and a duration are exactly what rate, errors and duration need. And a service-graph edge, because OBI knows both the caller and the peer address.
A note on AI traffic. The same idea now applies to LLM calls. OBI parses requests to OpenAI, Anthropic, Gemini and Bedrock, and MCP traffic, from the wire: model, token counts, tool calls. It is the same trick: the API is HTTP, the payload is JSON, and the TLS uprobe sees it in plaintext. If your team wants to know what its agents are spending without wrapping every client, this is a cheap first look.
5. Step 4: the hard part, context propagation
Everything so far gives you spans. A span is not a trace. A trace is spans joined by parent and child links, across services, and the only thing that joins them is a shared trace ID passed from one service to the next. In the OpenTelemetry world that ID travels in the traceparent header:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
│ └──────────── trace id ────────────┘ └─ parent span ─┘ └ flagsWith an SDK, your code reads the incoming header and writes it onto every outgoing call. With OBI, nobody touches your code, so something else has to write that header. Reading is the easy half: OBI parses an incoming traceparent automatically (on kernel 5.17 and newer). Writing is the hard half, and it is off by default. You have to turn it on.
There are four ways OBI can write context, and each one has a catch.
↳ four ways to write context, four catches
- HTTP headerplain HTTP
An sk_msg program inserts a traceparent header into the outgoing bytes.
+ Works with any OTel SDK on the other side.− Useless once the traffic is TLS: the header would sit inside the encryption. - TCP/IP packetHTTPS
The context goes into the packet header instead of the HTTP header.
+ Works without touching the encrypted payload.− Only OBI on the other side can read it. An L7 proxy or load balancer that ends the connection drops it. - gRPC stream headergRPC (HTTP/2)
A per-stream traceparent HPACK header.
+ Matches how gRPC multiplexes many calls on one connection.− Packet-level context is per connection, so it cannot tell streams apart. Hence the separate path. - Go libraryGo, HTTP or HTTPS
bpf_probe_write_user writes the header into the Go process’s memory before TLS.
+ A real header, even over HTTPS, that SDKs and proxies understand.− Needs CAP_SYS_ADMIN. Blocked when the kernel is in lockdown mode, which is typical with Secure Boot.
The Go library mode is worth a closer look, because of the helper it relies on. bpf_probe_write_user lets an eBPF program write into a user-space process's memory. The kernel prints a warning to the log whenever a program that uses it is loaded, and lockdown mode turns it off entirely. That is not OBI being reckless. It is the kernel being honest about how sharp this tool is. (The KubeCon NA 2024 talk "So You Want to Write Memory with eBPF?" is the best explanation I know of what can go wrong.)
Async work is its own problem. A trace follows a request through one thread of execution. Real code hands work off. OBI follows Go goroutines that a request starts, up to six levels deep. A Go channel handoff to a long-lived worker is different: the worker was not started by the request, so the follow-up span becomes a span link, not a child. Node.js, Java thread pools, Ruby Puma and Python uvloop each have their own specific support.
Try the combinations below. Every service is a Go service traced by OBI, so this is the best case. Note how quickly one user request becomes several traces.
↳ trace stitcher · simplified illustration
- gateway → orders✗ new tracenothing writes context into the outgoing request
- orders → billing✗ new tracenothing writes context into the outgoing request
what the trace backend shows
3 traces for one user request. Try turning HTTPS on with each mode, then add the proxy.
The demo proves the two cases that matter most. Here is the async endpoint in the gateway:
jobs := make(chan string, 128)
go worker(jobs, ordersURL) // started at boot, not by any request
http.HandleFunc("/checkout-async", func(w http.ResponseWriter, _ *http.Request) {
select {
case jobs <- ordersURL + "/orders":
w.WriteHeader(http.StatusAccepted)
default:
http.Error(w, "queue full", http.StatusServiceUnavailable)
}
})The request finishes with a 202. The worker's call to orders happens later, on a goroutine that has no request to inherit. In Jaeger, it is not a child of the /checkout-async span. And with propagation switched off, even the plain synchronous /checkout falls apart:
CP=none docker compose up -d obi # propagation off
./load.sh # gateway and orders now show up as separate tracesSame binaries, same traffic. The only thing that changed is whether an agent was allowed to write into the requests.
6. Step 5: export
The last step is the boring one, and that is a compliment. Events move from the ring buffer into the agent. The agent builds spans and metrics and sends them over OTLP to any backend: Jaeger in the demo, or a Collector, or a vendor. OBI can also run inside the OpenTelemetry Collector as a receiver, so you deploy one thing instead of two.
It also plays well with SDKs that are already there. If an application has already set traceparent on a request, OBI uses that value instead of making up its own. That makes a gradual mix possible: OBI everywhere, SDKs where you need more, and one trace across both.
The whole demo, from nothing to a trace:
ls /sys/kernel/btf/vmlinux # BTF must exist
docker compose up -d --build
./load.sh # mix of /checkout, ?fail=1 and /checkout-async
open http://localhost:16686 # Jaeger: pick service "gateway"
docker compose logs -f obi # spans printed as text, live↳ five things the demo proves
- 1
/checkoutOne trace: gateway server span, gateway client span, orders server span, Redis INCR. No SDK anywhere. - 2
/checkout?fail=1orders returns 500. The error status is on the spans, and the RED error rate moves, all from the wire. - 3
/checkout-asyncgateway answers 202 at once. The worker’s call to orders is not a child of that request. - 4
CP=nonePropagation off. The same traffic becomes separate gateway and orders traces. - 5
hey, OBI off vs onLatency and the agent’s own CPU, measured instead of quoted.
The full source is the obi-demo repo: two services, a tiny Redis client, a Dockerfile and a compose file. Every snippet in this post comes from it.
7. Where it falls short
Now the opinion section. Zero-code tracing is real, and it is useful. It is not free, and it is not complete.
No business context
OBI sees POST /orders 500. It does not see which order, which customer, which feature flag, or why. The wire does not carry that, so no amount of clever parsing will find it. The OBI docs say this themselves: use SDKs for custom spans and business events. A trace that says "something in orders failed" is better than nothing. It is not what you want at 3am.
Privilege
This is the list of Linux capabilities the agent can ask for, depending on what you turn on:
↳ what the agent asks for
- CAP_BPF load programs and maps
- CAP_PERFMON attach kprobes and uprobes, read the ring buffer
- CAP_SYS_PTRACE read /proc/<pid>/exe and the memory of other processes
- CAP_NET_RAW raw AF_PACKET sockets for socket filters
- CAP_CHECKPOINT_RESTORE resolve other processes’ executables
- CAP_DAC_READ_SEARCH read files it does not own
- CAP_NET_ADMIN context propagation: TC and sk_msg programs
- CAP_SYS_ADMIN Go library-level propagation, or when perf_event_paranoid demands it
- CAP_SYS_RESOURCE only on kernels older than 5.11
On many managed clusters it is root in practice. The default perf_event_paranoid setting on AKS and EKS nodes pushes you to CAP_SYS_ADMIN, which is most of root. The demo uses privileged: true to stay short. In a real cluster, list the capabilities explicitly and expect a security review. It is a fair question to ask of anything that can read every process's memory on the node.
It writes to your traffic
This is the point I would put in front of the person who has to approve the rollout. Context propagation means an agent is modifying requests in flight: adding headers, changing packets, or writing into process memory. When that code has a bug, the bug is in your traffic. OBI v0.13.0 fixed memory-safety bugs in exactly this path, bugs that could corrupt TLS connections. That is not a reason to avoid OBI. It is the honest reason to stage the rollout: observe-only first, propagation on a few services next, then wider.
The agent is attack surface
A privileged process that parses every byte on the node is a target. CVE-2026-45680, fixed in 0.9.0, let heavy traffic through instrumented services exhaust the agent's CPU. Nobody broke in, but a privileged agent pinned at 100% CPU on every node is its own kind of outage. Patch it like you patch the kernel, not like you patch a dashboard.
Kernel floor
Linux 5.8 or newer with BTF (RHEL-family 4.18 kernels with backports also work), on amd64 or arm64. Reading incoming headers for propagation needs 5.17. Most current distributions are fine. Older fleets and some custom kernels are not: the stock Raspberry Pi 5 kernel from the Beyond 65k post ships without BTF.
Version churn
It is still 0.x. Metric names and default labels have changed between releases. If you build dashboards and alerts on OBI's output today, pin the version and read the release notes before every upgrade.
Overhead: measure it yourself
There is no official benchmark. The docs say "minimal overhead", and vendors repeat it. Overhead depends on request rate, payload size, protocols and which features are on, so any single number would be misleading anyway. The demo includes a harness that measures it on your own hardware:
docker compose stop obi
hey -z 30s -c 50 http://localhost:8080/checkout > baseline.txt
docker compose start obi && sleep 10
hey -z 30s -c 50 http://localhost:8080/checkout > with-obi.txt
docker stats --no-stream $(docker compose ps -q obi)Compare p50 and p99, not the average, and look at the agent's own CPU. Then run the same test with propagation on and off, because writing context costs more than reading it.
8. My take: the floor, not the ceiling
Use eBPF instrumentation for:
- The baseline. Every service gets RED metrics, a service graph and basic traces on day one.
- Legacy and third-party services whose code you cannot or will not change.
- Fast coverage during a migration, or in a cluster nobody has instrumented yet.
Use SDKs for:
- Business spans: the order ID, the tenant, the step in the checkout that failed.
- Custom attributes that exist only in your code's variables.
- Async pipelines, queues and batch jobs, where the link between cause and effect lives in your code, not on the wire.
The likely end state is eBPF as the floor and SDKs as the ceiling. OBI's own roadmap points the same way, with hybrid instrumentation that wraps SDK spans instead of competing with them.
Having built eBPF agents myself, the lesson I keep relearning is that capturing data is the easy part. The kernel will happily give you more events than you can use. The hard part is always correlation: which event belongs to which request, across threads, processes and machines. OBI is no exception. Section 3 is the part that demos well. Section 5 is where the real engineering, and the real risk, lives.
Zero-code tracing is not magic. It is eBPF watching five places in your stack and writing in one of them.
So here is the question I would ask your team: would you let an agent rewrite your packets in prod to get free traces?
Related: TCP from the inside covers sk_buff, tcp_sendmsg and the kernel path that the kprobes in this post sit on. The anatomy of a lightweight monitoring agent's performance covers what it costs to run an agent on every node.
References: OBI docs · distributed traces · security and capabilities · releases · OBI 2026 goals · CVE-2026-45680