5 ms·

I might be a bit biased [0], but I feel like this is where profiling tools, and particularly continuous profiling tools shine. Sampling profilers look at stack

I might be a bit biased [0], but I feel like this is where profiling tools, and particularly continuous profiling tools shine. Sampling profilers look at stack traces X number of times per second resulting in a very low CPU overhead. And so essentially you're getting a very high precision trace. It's not exactly the same (e.g the order in which functions were called is not preserved), but can still be used for the same type of performance analysis. For rust you can use eBPF to get these traces down to system calls. You can even profiler other people's software with it. [0] I'm a co-founder at Pyroscope, where we're building an open source continuous profiling platform. https://pyroscope.io/ https://pyroscope.io/

pkhuong 5y ago
Pure sampling profiles are less helpful when solving latency rather than throughput problems, especially so for rare (e.g. p99) latency spikes.
NightMKoder 5y ago
I’ll have to disagree with you - the problems are similar but somewhat unrelated. Sampling profilers traditionally only sample threads that are actively doing something - i.e. using CPU time. Tracing is closer to wall time profiling rather than CPU time - and global instrumentation actually sucks at that kind of enumeration. The overhead of iterating through 100k goroutines would be too high at any reasonable frequency when most of them are just blocked on something. And most are probably blocked on something you don’t even care to see (eg a worker waiting for work). Aside from the sampling issue there is also the problem of non-blocking frameworks/languages. It’s easy to add tracing to JS functions that use async/await but it is definitely a mini research project to recover stacks from in-flight promise state machines. Most languages fall into this bucket unfortunately. Now all that said, if you can make some automation that preemptively creates spans for you - kind of like a sampling profiler “creates” stack traces - that would be super cool. The worst part about tracing is sprinkling the tracing code in all the places you need it. You inevitably miss a few spots that show up as “missing” bars in the gannt view. Having something that fills in those blanks is invaluable.
- MrBuddyCasino 5y ago
  > The worst part about tracing is sprinkling the tracing code in all the places you need it. On the JVM this is usually done by agents that automatically attach to your process, you don't have to instrument your code at all. I think Datadog open-sourced their implementation?
  - vips7L 5y ago
    Java Flight Recorder is basically 0 overhead as well and built directly into the VM.
    - MrBuddyCasino 5y ago
      Isn't Flight Recorder profiling and not tracing? Or has this been added?
    - NightMKoder 5y ago
      Both JFR and agent based implementations only work with non async code. As soon as you do async anything, stack traces are no longer representative so automatic instrumentation is useless. That said, I’m really hoping Project Loom will fix that issue by getting rid of most async callback-style I/O.
zerd 5y ago
Tracing is at a different level of detail. For instance you can see which user and specific query is missing caches and spending a lot of time waiting for IO. Unless you inspect local variables while walking the stack I don't see how you'd get that just just profiling. But a combination would be preferable (some observability tools have started on this, e.g. the profiler will tell the tracer where add additional instrumentation).

pkhuong 5y ago

Pure sampling profiles are less helpful when solving latency rather than throughput problems, especially so for rare (e.g. p99) latency spikes.

NightMKoder 5y ago

I’ll have to disagree with you - the problems are similar but somewhat unrelated. Sampling profilers traditionally only sample threads that are actively doing something - i.e. using CPU time. Tracing is closer to wall time profiling rather than CPU time - and global instrumentation actually sucks at that kind of enumeration. The overhead of iterating through 100k goroutines would be too high at any reasonable frequency when most of them are just blocked on something. And most are probably blocked on something you don’t even care to see (eg a worker waiting for work). Aside from the sampling issue there is also the problem of non-blocking frameworks/languages. It’s easy to add tracing to JS functions that use async/await but it is definitely a mini research project to recover stacks from in-flight promise state machines. Most languages fall into this bucket unfortunately. Now all that said, if you can make some automation that preemptively creates spans for you - kind of like a sampling profiler “creates” stack traces - that would be super cool. The worst part about tracing is sprinkling the tracing code in all the places you need it. You inevitably miss a few spots that show up as “missing” bars in the gannt view. Having something that fills in those blanks is invaluable.

MrBuddyCasino 5y ago
> The worst part about tracing is sprinkling the tracing code in all the places you need it. On the JVM this is usually done by agents that automatically attach to your process, you don't have to instrument your code at all. I think Datadog open-sourced their implementation?
- vips7L 5y ago
  Java Flight Recorder is basically 0 overhead as well and built directly into the VM.
  - MrBuddyCasino 5y ago
    Isn't Flight Recorder profiling and not tracing? Or has this been added?
  - NightMKoder 5y ago
    Both JFR and agent based implementations only work with non async code. As soon as you do async anything, stack traces are no longer representative so automatic instrumentation is useless. That said, I’m really hoping Project Loom will fix that issue by getting rid of most async callback-style I/O.

zerd 5y ago

Tracing is at a different level of detail. For instance you can see which user and specific query is missing caches and spending a lot of time waiting for IO. Unless you inspect local variables while walking the stack I don't see how you'd get that just just profiling. But a combination would be preferable (some observability tools have started on this, e.g. the profiler will tell the tracer where add additional instrumentation).