5 ms·
How do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to se
by avita1 3y ago
How do you solve the context propagation issue with eBPF based instrumentation?
E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program sees the same request ID.
IME that's where most of the overhead (and value) from a manual tracing library comes from.
- edenfed 3y agoIt depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.
- camel_gopher 3y agoThat’s a very big assumption, at least for Go based applications.
- edenfed 3y agoWe also thinking on implementing fallback mechanism to automatically propagate context on the same goroutine if context.Context is not passed
- nulld3v 3y agoI don't think it's unreasonable, you need a Context to make a gRPC call and you get one when handling a gRPC call. It usually doesn't get lost in between.
- otterley 3y agoTrue for gRPC, but not necessarily for HTTP - the HTTP client and server packages that ship with Go predate the Context package by quite a long while.
- spullara 3y agoGoing to be rough for supporting virtual threads then?
- edenfed 3y agoWe have a solution for virtual thread as well. Currently working on a blog post describing exactly how. Will update once releases
- marwis 3y agoScopedValue solves that problem: https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/ScopedValue.html https://docs.oracle.com/en/java/javase/21/docs/api/java.base...
- rocmcd 3y ago100%. Context propagation is _the_ key to distributed tracing, otherwise you're only seeing one side of every transaction. I was hoping odigos was language/runtime-agnostic since it's eBPF-based, but I see it's mentioned in the repo that it only supports: > Java, Python, .NET, Node.js, and Go Apart from Go (that is a WIP), these are the languages already supported with Otel's (non-eBPF-based) auto-instrumentation. Apart from a win on latency (which is nice, but could in theory be combated with sampling), why else go this route?
- edenfed 3y agoeBPF instrumentation does not require code changes, redeployment or restart to running applications. We are constantly adding more language support for eBPF instrumentation and are aiming to cover the most popular programming languages soon. Btw, not sure that sampling is really the solution to combat overhead, after all you probably do want that data. Trying to fix production issue when the data you need is missing due to sampling is not fun
- rocmcd 3y agoAll good points, thank you. What's the limit on language support? Is it theoretically possible to support any language/runtime? Or does it come down to the protocol (HTTP, gRPC, etc) being used by the communicating processes?
- edenfed 3y agoWe already solved compiled languages (Go, C, Rust) and JIT languages (Java, C#). Interpreted languages (Python, JS) are the only ones left, hopefully we will solve these as well soon. The big challenge is supporting all the different runtimes, once that is solved implementing support for different protocols / open-source libraries is not as complicated.
- jetbalsa 3y agoGot to get PHP on that list :)
- RonFeder 3y agoThe eBPF programs handle passing the context through the requests by adding a field to the header as you mentioned. The injected field is according to the w3c standard.