3 ms·

Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wi

by jeffbee 12d ago

Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.

MomsAVoxell 12d ago
If you’re not using eBPF to trace your app you’re doing it wrong.
- jeffbee 12d ago
  The low cost of eBPF tracing is another myth.
  - MomsAVoxell 12d ago
    1) Its no myth, but you can definitely foot-bullet into doing it wrong, and 2) it's a far better path to take than in-app telemetry.
  - deleted 12d ago
    [deleted]
- MobiusHorizons 11d ago
  Doesn’t that only work on Linux? And then only for things that make syscalls? Presumably people have to trace other slow paths sometime.
nicoburns 12d ago
One legitimately great thing about LLMs is that it makes it feasible to add these kind of tracing instrumentations temporarily for profiling and then throw them away so they never reach source control let alone production.
- jeffbee 12d ago
  I can get an LLM to trace my incomprehensible Tokio application which was also written by an LLM, which is why I don't understand its behavior. Truly the future we were promised.
  - brunoarueira 11d ago
    I guess you should adopt RFCs or ADRs to help clarify the Tokio application, like this https://github.com/brunoarueira/thoth-mesh/tree/main/docs/adr https://github.com/brunoarueira/thoth-mesh/tree/main/docs/ad.... This project is vibe coded, but I had put the effort to create issues, roadmap and ADRs, so later I can understand the project without going deep on the code!
- shim__ 11d ago
  Reaching source control is fine as long as there is a compile time flag to disable the whole thing, which tokio-tracing does
Veserv 12d ago
That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload. You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything, when you get to the 10-30% range. The only way you get to “majority” is if your trace implementation is slower than time travel debugging and provides less information, but then why choose something worse in every dimension.
- jeffbee 12d ago
  I'm just reporting from the trenches here. I think you are suggesting that everyone is aware of and capable of using state-of-the-art (from 20 years ago) tracing schemes like XRay[1], when in reality they are not. Most projects would be well-served by any basic profiler but even profiling is apparently for wizards, because I've seen a lot of projects that will resort to manually annotating functions with OTel trace spans, which are ~millions of times more expensive than function calls. Even eBPF uprobe/uretprobe is 100x more expensive than XRay, at a minimum. HotSpot's JFR is like a miracle compared to what people suffer through to diagnose Rust+Tokio. 1: https://llvm.org/docs/XRay.html https://llvm.org/docs/XRay.html ... is there even a Rust analog to this?
  - duped 12d ago
    Not even an analog: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/instrument-xray.html https://doc.rust-lang.org/beta/unstable-book/compiler-flags/... It's worth pointing out though that just tracing function calls isn't good enough for the kinds of stackless coroutines that run in async Rust tasks. You need a way of mapping between the async tasks and the compiler emitted traces. afaik, C/C++ have the same problem.
    - jeffbee 12d ago
      The difference is nobody in the C++ community believes that a dominant asynchronous executor library exists, and there is not a pervasive belief that it would be helpful.
foota 12d ago
Just curious, why? Is this true even if you did something like a per-CPU histogram that uses atomic ops to increment?
- jeffbee 12d ago
  If you have a per-cpu metric there would not be a reason to use atomic instructions to mutate it.
  - loeg 11d ago
    In general your unpinned userspace threads will hit the same CPU 99.99% of the time, but not 100%.
    - jeffbee 11d ago
      Sure. You get the pointer, you lock the mutex, 99.99% of the time that is uncontended, then you set all the metrics and release it.
      - loeg 11d ago
        Taking the mutex uses (uncontended) atomic ops.
rusbus 12d ago
Was this in a specific application? I wouldn't necessarily expect that histogram to be particularly bad for most applications.
- jeffbee 12d ago
  Reading the clock every time you jump into a closure is in fact incredibly wasteful, and is exacerbated by chopping work up into tiny chunks for questionable reasons.
hansvm 11d ago
I'm not sure how other people are using LLMs for instrumentation, but IMO the layer you want running in prod is very different from what you want running for a one-off test. E.g., I have some code floating around which burns a pinned core on increasing a counter, with a little wrapper code around grabbing real timestamps at the beginning and end of a session and converting between the two units of time. It's helpful when microbenchmarking a very small unit of code as it actually behaves in a larger program (not perfect -- obviously tweaks the icache and pipeline behavior at a minimum -- but no measurement has zero tradeoffs, and you're always choosing which set of tradeoffs you prefer). An LLM can quickly instrument the call path I care about while I study this or that intervention. The ability to bang out a large amount of throwaway code is delightful.

by jeffbee 12d ago

Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.

MomsAVoxell 12d ago

If you’re not using eBPF to trace your app you’re doing it wrong.

jeffbee 12d ago
The low cost of eBPF tracing is another myth.
- MomsAVoxell 12d ago
  1) Its no myth, but you can definitely foot-bullet into doing it wrong, and 2) it's a far better path to take than in-app telemetry.
- deleted 12d ago
  [deleted]
MobiusHorizons 11d ago
Doesn’t that only work on Linux? And then only for things that make syscalls? Presumably people have to trace other slow paths sometime.

nicoburns 12d ago

One legitimately great thing about LLMs is that it makes it feasible to add these kind of tracing instrumentations temporarily for profiling and then throw them away so they never reach source control let alone production.

jeffbee 12d ago
I can get an LLM to trace my incomprehensible Tokio application which was also written by an LLM, which is why I don't understand its behavior. Truly the future we were promised.
- brunoarueira 11d ago
  I guess you should adopt RFCs or ADRs to help clarify the Tokio application, like this https://github.com/brunoarueira/thoth-mesh/tree/main/docs/adr https://github.com/brunoarueira/thoth-mesh/tree/main/docs/ad.... This project is vibe coded, but I had put the effort to create issues, roadmap and ADRs, so later I can understand the project without going deep on the code!
shim__ 11d ago
Reaching source control is fine as long as there is a compile time flag to disable the whole thing, which tokio-tracing does

Veserv 12d ago

That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload. You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything, when you get to the 10-30% range. The only way you get to “majority” is if your trace implementation is slower than time travel debugging and provides less information, but then why choose something worse in every dimension.

jeffbee 12d ago
I'm just reporting from the trenches here. I think you are suggesting that everyone is aware of and capable of using state-of-the-art (from 20 years ago) tracing schemes like XRay[1], when in reality they are not. Most projects would be well-served by any basic profiler but even profiling is apparently for wizards, because I've seen a lot of projects that will resort to manually annotating functions with OTel trace spans, which are ~millions of times more expensive than function calls. Even eBPF uprobe/uretprobe is 100x more expensive than XRay, at a minimum. HotSpot's JFR is like a miracle compared to what people suffer through to diagnose Rust+Tokio. 1: https://llvm.org/docs/XRay.html https://llvm.org/docs/XRay.html ... is there even a Rust analog to this?
- duped 12d ago
  Not even an analog: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/instrument-xray.html https://doc.rust-lang.org/beta/unstable-book/compiler-flags/... It's worth pointing out though that just tracing function calls isn't good enough for the kinds of stackless coroutines that run in async Rust tasks. You need a way of mapping between the async tasks and the compiler emitted traces. afaik, C/C++ have the same problem.
  - jeffbee 12d ago
    The difference is nobody in the C++ community believes that a dominant asynchronous executor library exists, and there is not a pervasive belief that it would be helpful.

foota 12d ago

Just curious, why? Is this true even if you did something like a per-CPU histogram that uses atomic ops to increment?

jeffbee 12d ago
If you have a per-cpu metric there would not be a reason to use atomic instructions to mutate it.
- loeg 11d ago
  In general your unpinned userspace threads will hit the same CPU 99.99% of the time, but not 100%.
  - jeffbee 11d ago
    Sure. You get the pointer, you lock the mutex, 99.99% of the time that is uncontended, then you set all the metrics and release it.
    - loeg 11d ago
      Taking the mutex uses (uncontended) atomic ops.

rusbus 12d ago

Was this in a specific application? I wouldn't necessarily expect that histogram to be particularly bad for most applications.

jeffbee 12d ago
Reading the clock every time you jump into a closure is in fact incredibly wasteful, and is exacerbated by chopping work up into tiny chunks for questionable reasons.

hansvm 11d ago

I'm not sure how other people are using LLMs for instrumentation, but IMO the layer you want running in prod is very different from what you want running for a one-off test. E.g., I have some code floating around which burns a pinned core on increasing a counter, with a little wrapper code around grabbing real timestamps at the beginning and end of a session and converting between the two units of time. It's helpful when microbenchmarking a very small unit of code as it actually behaves in a larger program (not perfect -- obviously tweaks the icache and pipeline behavior at a minimum -- but no measurement has zero tradeoffs, and you're always choosing which set of tradeoffs you prefer). An LLM can quickly instrument the call path I care about while I study this or that intervention. The ability to bang out a large amount of throwaway code is delightful.