5 ms·

Tracing Goroutines in Realtime with eBPF

80 points by darccio 6mo ago

Daviey 6mo ago
[dead]
xtoilette 6mo ago
Cool! is there something similar for async rust ?
- fch42 6mo ago
  https://docs.rs/tokio-console/latest/tokio_console/ https://docs.rs/tokio-console/latest/tokio_console/ comes to mind. I'm not aware of eBPF-based dynamic instrumentation for either tokio or other async runtimes, but I'm also not (in need to be) debugging this on a daily basis.
broken_broken_ 6mo ago
Interesting to see an ebpf approach to this idea, some time ago I did the same with Dtrace: https://gaultier.github.io/blog/detecting_goroutine_leaks_with_dtrace.html https://gaultier.github.io/blog/detecting_goroutine_leaks_wi... However I did not think of observing the ‘castgstatus’ runtime function. Always nice to be able to compare the DTrace and Ebpf approaches, they both have different strong points.
tsuzu 6mo ago
So cool! Does it support multiple Go versions?
- flakes 6mo ago
  I would say that this is limited to a targeted Go version and architecture. For example, the filter checks for the goroutine pointer on `r28` which is correct for arm64 but not universally true. Any changes to the struct layouts, stack, or heap layouts would also cause failures in these lookups. E.g., in Go 1.17, many functions now use direct register mappings for arguments rather than always placing arguments in the stack. Would need to thoroughly vet compatability with each new Go version before using something like this in production.
trueno 6mo ago
that is actually sick. how common is it for go devs to experience leaking goroutines ? id like to think go is a lot less shoot yourself in the foot here since they provide a framework for concurrency/parallelism rather than you working with the tiny pieces of it and building out the architecture yourself, but ive only needed to use goroutines once and it was a pretty problem-free experience.
- usrnm 6mo ago
  > how common is it for go devs to experience leaking goroutines About as often as leaking memory in C++
  - trueno 6mo ago
    F. again i have minimal experience actually ever needing it in go, but guessing this is just generally the exercise of managing the lifecycle of a goroutine well in your code? proper handling so things dont get orphaned in buffer, fire and forget woopsies, etc. early on i do feel like go kinda advertised batteries included concurrency but i kinda wished they advertised the foot-shooting-mechanisms and gaps in the abstraction a little more. overall i prefer to have enough control to choose how to manage the lifecycle. mem leaks bum me out and kill my steam, at least from my experience with c/cpp.
- broken_broken_ 6mo ago
  It depends how much the code uses manually spawned goroutines, and how complex the lifecycle of these goroutines is… in big codebases such as kubernetes, docker, etc, it has been a problem. There have been research papers and blogs about this, but most Go developers are not aware of this issue it seems.
darshil2023 6mo ago
[dead]

80 points by darccio 6mo ago

Daviey 6mo ago

[dead]

xtoilette 6mo ago

Cool! is there something similar for async rust ?

fch42 6mo ago
https://docs.rs/tokio-console/latest/tokio_console/ https://docs.rs/tokio-console/latest/tokio_console/ comes to mind. I'm not aware of eBPF-based dynamic instrumentation for either tokio or other async runtimes, but I'm also not (in need to be) debugging this on a daily basis.

broken_broken_ 6mo ago

Interesting to see an ebpf approach to this idea, some time ago I did the same with Dtrace: https://gaultier.github.io/blog/detecting_goroutine_leaks_with_dtrace.html https://gaultier.github.io/blog/detecting_goroutine_leaks_wi... However I did not think of observing the ‘castgstatus’ runtime function. Always nice to be able to compare the DTrace and Ebpf approaches, they both have different strong points.

tsuzu 6mo ago

So cool! Does it support multiple Go versions?

flakes 6mo ago
I would say that this is limited to a targeted Go version and architecture. For example, the filter checks for the goroutine pointer on `r28` which is correct for arm64 but not universally true. Any changes to the struct layouts, stack, or heap layouts would also cause failures in these lookups. E.g., in Go 1.17, many functions now use direct register mappings for arguments rather than always placing arguments in the stack. Would need to thoroughly vet compatability with each new Go version before using something like this in production.

trueno 6mo ago

that is actually sick. how common is it for go devs to experience leaking goroutines ? id like to think go is a lot less shoot yourself in the foot here since they provide a framework for concurrency/parallelism rather than you working with the tiny pieces of it and building out the architecture yourself, but ive only needed to use goroutines once and it was a pretty problem-free experience.

usrnm 6mo ago
> how common is it for go devs to experience leaking goroutines About as often as leaking memory in C++
- trueno 6mo ago
  F. again i have minimal experience actually ever needing it in go, but guessing this is just generally the exercise of managing the lifecycle of a goroutine well in your code? proper handling so things dont get orphaned in buffer, fire and forget woopsies, etc. early on i do feel like go kinda advertised batteries included concurrency but i kinda wished they advertised the foot-shooting-mechanisms and gaps in the abstraction a little more. overall i prefer to have enough control to choose how to manage the lifecycle. mem leaks bum me out and kill my steam, at least from my experience with c/cpp.
broken_broken_ 6mo ago
It depends how much the code uses manually spawned goroutines, and how complex the lifecycle of these goroutines is… in big codebases such as kubernetes, docker, etc, it has been a problem. There have been research papers and blogs about this, but most Go developers are not aware of this issue it seems.

darshil2023 6mo ago

[dead]