3 ms·
I think this is great work. Hopefully this can motivate upstream to reconsider the proposal for it. In the long run I don't think anybody wants to run with a fo
by felixge 5y ago
I think this is great work. Hopefully this can motivate upstream to reconsider the proposal for it. In the long run I don't think anybody wants to run with a forked version of Go for better profiling.
> The ability to monitor go programs with a very high sampling frequency — up to 10s of microseconds
I'm curious about the overhead of this and will probably try to measure. This work certainly overcomes the limitations of setitimer(2) for sampling rate, but faster sampling is going to increase the performance overhead significantly.
In particular Go's built-in stack unwinding APIs use gopclntab (which is Go's Plan9 inspired version of eh_frame). This is rather slow compared to frame pointer unwinding (up to 55x in my testing). I think on average you can expect to spend at least 1 microsecond on unwinding, and then some additional time hashing the stack and updating the in-memory profile (perhaps another microsecond? I haven't measured this part yet). I've started writing some of this up here [1] but it's still WIP.
Anyway, based on the above, sampling every 10usec will likely slow the target program by ~20%. This is probably a very optimistic estimate because it doesn't take into account all the CPU cache thrashing that will happen.
As I said, I'll try to take a closer look later, but if anybody has some insights/comments here I'd be very interested.
[1] https://github.com/DataDog/go-profiler-notes/blob/main/stack-traces.md https://github.com/DataDog/go-profiler-notes/blob/main/stack...
Disclaimer: I work at Datadog on Continuous Go Profiling, but I'm here speaking for myself.
- akshayshah 5y agoI suspect you're right about overhead. 10us seems much more aggressive than even the author is actually suggesting, though - the motivating example was a service with latencies in the 10s of ms, so sampling every few ms would be sufficient. Also, if anyone reading isn't familiar with sampling profilers, Felix gave a great talk explaining them: https://github.com/felixge/talks/blob/master/fgprof/ https://github.com/felixge/talks/blob/master/fgprof/
- felixge 5y agoYeah, decent sampling for requests that last 10s of ms seems absolutely feasible with this approach, unlike the current setitimer(2) stuff. I'm really looking forward to taking it for a spin soon and see how far the sampling rate can be pushed without introducing noticeable overhead in practice!