8 ms·
eBPF-based auto-instrumentation outperforms manual instrumentation
- nevodavid10 3y agoThis is great. Can you elaborate on how the performance is better?
- Barakikia 3y agoOur focus was on latency. The reason we were able to cut it down was due to the fact that eBPF-based automatic instrumentation separates the recording from the processing.
- grazio 3y agoHow did you actually reduce the latency here ?
- RonFeder 3y agoThe main factor for reduced latency is the separation between recording and processing of data. The eBPF programs are the only overhead for the instrumented process in terms of latency. The eBPF programs transfer the collected data to a separate process which handles all the exporting. In contrast to manually adding code to an application which adds latency and memory footprint in terms of handling the exported data.
- chii 3y agobut the processing will still cost CPU time which takes it away from the 'main' process, unless it's transferred away from the machine and processed elsewhere. Unless if eBPF can do such processing much more efficiently than the application's own code, i don't see how it reduces latency differently from a properly threaded app. Of course, using eBPF makes an app instrument-able without changes is good enough a reason to use it.
- RonFeder 3y agoCompared to multi-threaded process, there is still a big advantage in terms of latency. Handling all the exporting in the same process will greatly effect GC operations which require stop the world handling.
- CSDude 3y agoSomewhat related, I mainly code in Kotlin. Adding open telemetry was just adding agent to command line args (usual Java/JVM magic most people don't like). Then I had a project in Go and I got so tired of all the steps it took (setup and ensuring each context is instrumented) and just gave up. We still add our manual instrumentation for customization, but auto-instrumentation made adoption much easier in the day 0.
- edenfed 3y agoI think eBPF has also great potential to help JVM-based languages. Especially around performance aspects even comparing to the current java agents which use bytecode manipulation.
- marwis 3y agoThe article mentions avoiding GC pressure and separation between recording and processing as big wins for performance for runtimes like Java but you could do the same inside Java by using ring buffer, no?
- edenfed 3y agoInteresting idea. I think that as long as you able to do processing, serializing and delivery in other process and save this work from your application runtime you should see great performance
- remoquete 3y agoOTel autoinstrumentation is in the works, check it out: https://github.com/open-telemetry/opentelemetry-go-instrumentation#get-started https://github.com/open-telemetry/opentelemetry-go-instrumen... (I wrote and tested that guide).
- avita1 3y agoHow do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program sees the same request ID. IME that's where most of the overhead (and value) from a manual tracing library comes from.
- edenfed 3y agoIt depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.
- camel_gopher 3y agoThat’s a very big assumption, at least for Go based applications.
- edenfed 3y agoWe also thinking on implementing fallback mechanism to automatically propagate context on the same goroutine if context.Context is not passed
- nulld3v 3y agoI don't think it's unreasonable, you need a Context to make a gRPC call and you get one when handling a gRPC call. It usually doesn't get lost in between.
- otterley 3y agoTrue for gRPC, but not necessarily for HTTP - the HTTP client and server packages that ship with Go predate the Context package by quite a long while.
- heyeb_25 3y agoIf I am manually implemented all my logs, what do I need to do to move to Odgios?
- edenfed 3y agoNothing special, if you are working on Kubernetes its as easy as running `odigos install` CLI and pointing to your current monitoring system.
- bakery_bake 3y agoAccording to what you say, nobody should implement logs manually? I will check Odigos.
- edenfed 3y agoLogs are easy and familiar API for adding additional data to your traces. They still have their place, Odigos is just adding much more context.
- jrockway 3y agoThey don't really show any of the settings they used, but for traces, I imagine if you have a reasonable sampling rate, then you aren't going to be running any code for most requests, so it won't increase latency. (Looking at their chart, I guess they are sampling .1% of requests, since 99.9% is where latency starts increasing. I am not sure if I would trace .1% of pages loads to google.com, as their table implies. Rather, I'd pick something like 1 request per second, so that latency does not increase as load increases.) A lot of Go metrics libraries, specifically Prometheus, introduce a lot of lock contention around incrementing metrics. This was unacceptably slow for our use case at work and I ended up writing a metrics system that doesn't take any locks for most cases. (There is the option to introduce a lock for metrics that are emitted on a timed basis; i.e. emit tx_bytes every 10s or 1MiB instead of at every Write() call. But this lock is not global to the program; it's unique to the metric and key=value "fields" on the metric. So you can have a lot of metrics around and not content on locks.) The metrics are then written to the log, which can be processed in real time to synthesize distributed traces and prometheus metrics, if you really want them: https://github.com/pachyderm/pachyderm/blob/master/src/internal/meters/meters.go#L1 https://github.com/pachyderm/pachyderm/blob/master/src/inter... (Our software is self-hosted, and people don't have those systems set up, so we mostly consume metrics/traces in log form. When customers have problems, we prepare a debug bundle that is mostly just logs, and then we can further analyze the logs on our side to see event traces, metrics, etc.) As for eBPF, that's something I've wanted to use to enrich logs with more system-level information, but most customers that run our software in production aren't allowed to run anything as root, and thus eBPF is unavailable to them. People will tolerate it for things like Cilium or whatever, but not for ordinary applications that users buy and request that their production team install for them. Production Linux at big companies is super locked down, it seems, much to my disappointment. (Personally, my threat model for Linux is that if you are running code on the machine, you probably have root through some yet-undiscovered kernel bug. Historically, I've been right. But that is not the big companies' security teams' mental model, it appears. They aren't paranoid enough to run each k8s pod in a hypervisor, but are paranoid enough to prevent using CAP_SYS_ADMIN or root.)
- edenfed 3y agoThanks for the valuable feedback! We used a constant throughout of 10,000 rps. The exact testing setup can be found under “how we tested”. I think the example you gave for the lock used by Prometheus library is a great example why generation of traces/metrics is a great fit for offloading to different process (an agent). Patchyderm looks very interesting however I am not sure how you can generate distributed traces based on metrics, how do you fill in the missing context propagation? Our way to deal with eBPF root requirements is to be transparent as possible. This is why we donated the code to the CNCF and developing as part of the OpenTelemetry community. We hope that being open will make users trust us. You can see the relevant code here: https://github.com/open-telemetry/opentelemetry-go-instrumentation https://github.com/open-telemetry/opentelemetry-go-instrumen...
- otterley 3y agoThe column in the table claiming the "number of page loads that would experience the 99th %ile" is mathematically suspect. It directly contradicts what a percentile is. By definition, at 99th percentile, if I have 100 page loads, the one with the worst latency would be over the 99th percentile. That's not 85.2%, 87.1%, 67.6%, etc. The formula shown in that column makes no sense at all.
- edenfed 3y agoI recommend watching Gil Tene’s talk, I think he explains the math better than I do: https://www.youtube.com/watch?v=lJ8ydIuPFeU https://www.youtube.com/watch?v=lJ8ydIuPFeU
- tpankaj 3y agoThat's not what that column is supposed to mean afaict. The way I read it is it's showing that if the website requires hundreds of different parallel backend service calls to serve the page load, what's the probability a page load hits the p99 instrumentation latency? We have a similar chart at my job to illustrate the point that high p99 latency on a backend service doesn't mean only 1% of end-user page loads are affected.
- otterley 3y agoAh, I see. So, for example, if one page request would result in 190 different backend requests to fulfill, then the possibility that at least one of those subrequests exceeds the 99th percentile would be 85.2%. That makes a lot more sense.
- bjt12345 3y agoBut what if the 100 page loads are just a sample of the population?
- chabad360 3y agoHow hard is it to use Odigos without k8s? We mainly use docker compose for our deployments (because it's convenient, and we don't need scale), but I'm having trouble finding anything in the documentation that explains the mechanism for hooking into the container (and hence I have no clue how to repurpose it).
- edenfed 3y agoWe are currently supporting just Kubernetes environments. docker-compose, VMs, and Serverless are on our roadmap and will be ready soon
- ranting-moth 3y agoWebsite doesn't display correctly on FF on android. Text bleeds on left and right side.
- edenfed 3y agoThank you for reporting will fix ASAP
- zengid 3y agoAnyone from the dtrace community want to enlighten a n00b about how eBPF compares to what dtrace does?
- zengid 3y agoFrom the hot takes in this post from 2018 [0], I may be asking a contentious question. [0] https://news.ycombinator.com/item?id=16375938 https://news.ycombinator.com/item?id=16375938
- edenfed 3y agoI don’t have a lot of experience using dtrace, but AFAIK the big advantage of eBPF over dtrace is that you do not need to instrument your application with static probes during coding.
- tanelpoder 3y agoDTrace (on Solaris at least) can instrument any userspace symbol or address, no need for static tracepoints in the app. One problem that DTrace has is that the "pid" provider that you use for userspace app tracing only works on processes that are already running. So, if more processes with the executable of interest launch after you've started DTrace, its pid provider won't catch the new ones. Then you end up doing some tricks like tracking exec-s of the binary and restarting your DTrace script...
- bcantrill 3y agoThat's not exactly correct, and is merely a consequence of the fact that you are trying to use the pid provider. The issue that you're seeing is that pid probes are created on-the-fly -- and if you don't demand that they are created in a new process, they in fact won't be. USDT probes generally don't have this issue (unless they are explicitly lazily created -- and some are). So you don't actually need/want to restart your DTrace script, you just want to force probes to be created in new processes (which will necessitate some tricks, just different ones).
- Thaxll 3y agoOf course it outperforms it, but it's basic instrumentation, how do you properly select the labels for example? In your application you will have custom instrumentation for business logic, so what do you do? Now you have two systems instrumenting the same app?
- edenfed 3y agoYou can enrich the spans created by eBPF by using OpenTelemetry APIs as usual, the eBPF instrumentation is a replacement for the instrumentation SDK. The eBPF program will detect the data recorded via the APIs and will add it to the final trace combining both automatic and manually created data.
- Andrew018 3y ago[dead]
- nikolay_sivko 3y agoDisclaimer: I'm a co-founder of Coroot. We're currently benchmarking our eBPF-based agent to measure its performance impact. Could you please elaborate on a few more details about your benchmark? - Did you measure the CPU usage of the eBPF agent? - How does Odigos handle eBPF's perfmap overflow, and did you measure any lost events between the kernel and the agent?
- anuraaga 3y agoIn the age of supply chain attack weariness and general risk skyrocketing, it is a bit funny seeing the many observability vendors wanting you to give them kernel mode access. And it's sad that most apps that will be most in need of automatic instrumentation are "frozen" rarely developed / updated apps at critical institutions like banks. As for the original post, opentelemetry is forced to be relatively slow because of a huge amount of semantic conventions that are meant to make data more useful. I won't go into the legitimacy of that, but while I haven't been able to verify the data this solution records, it is very unlikely to be recording as much information. Manual instrumentation would never loose to eBPF in principle, at least in a compiled language like Go, but eBPF does have great potential to perform better than OTel while recording far less data. Then comes blog post, users giving the keys to their kernel, and data ending up in the hands of an enemy state. I doubt that's the case this time but it's only a matter of time. Banking apps if you see this, please just instrument your code. Thank you.
- shivawu 3y agoHow does it work with nodejs? Iirc they don’t support ebpf
- himanshu2998 3y agocan we add manual spans (at service level) also as part of automated traces (at eBPF auto instrumentation) created by this approach? like can we access context in a running application? or will there be any sort of "traceparent" header present in incoming request?