7 ms·

looks very promising, one of the biggest issue in golang for me is profiling and constant memory leaks/pressure. Not sure if there is an alternative of what peo

looks very promising, one of the biggest issue in golang for me is profiling and constant memory leaks/pressure. Not sure if there is an alternative of what people use now

felixge 1y ago
I'd love to hear more! What kind of profiling issues are you running into? I'm assuming the inuse memory profiles are sometimes not good enough to track down leaks since they only show the allocation stack traces? Have you tried goref [1]?. What kind of memory pressure issues are you dealing with? [1] https://github.com/cloudwego/goref https://github.com/cloudwego/goref Disclaimer: I work on continuous profiling for Datadog and contribute to the profiling features in the runtime.
- tsimionescu 1y ago
  I for one am still mystified how it's possible that a GC language can't expose the GC roots in a memory profile. I've lost so many hours of my life manually trying to figure out what might be keeping some objects live, information the GC figures out every single time it runs...
  - felixge 1y ago
    Do you think the GC roots alone (goroutine stacks with goroutine id, package globals) would be enough? I think in many cases you'd want the reference chains. The GC could certainly keep track of those, but at the expense of making things slower. My colleagues Nick and Daniel prototyped this at some point [1]. Alternatively the tracing of reference chains can be done on heap dumps, but it requires maintaining a partial replica of the GC in user space, see goref [2] for that approach. So it's not entirely trivial, but rest assured that it's definitely being considered by the Go project. You can see some discussions related to it here [3]. Disclaimer: I contribute to the Go runtime as part of my job at Datadog. I can't speak on behalf of the Go team. [1] https://go-review.googlesource.com/c/go/+/552736 https://go-review.googlesource.com/c/go/+/552736 [2] https://github.com/cloudwego/goref/blob/main/docs/principle.md https://github.com/cloudwego/goref/blob/main/docs/principle.... [3] https://github.com/golang/go/issues/57175 https://github.com/golang/go/issues/57175
- defraudbah 1y ago
  no, haven't heard of goref yet but will give it a shot! usually I go with pprof, like basic stuff and it helps. I would NOT say memory leak is the biggest or most common issue I see, however as time goes and services become more complicated what I often see in the metrics is how RAM gets eaten and does not get freed as time goes, so the app eats more and more memory as time goes and only restart helps. It's hard to call it memory leak in "original meaning of memory leak" but the memory does not get cleaned up because the choices I made and I want to understand how to make it better. Thanks for the tool!
  - prerok 1y ago
    Sorry if this is a basic question but are you setting the GOMEMLIMIT? Also, are you running the code in a container? In K8s?
pjmlp 1y ago
Ideally, they would have learnt from other languages, and offered explicit control over what goes into the stack instead of relying into escape analysis alone. As it is, the only way to currently handle that is with " -gcflags -m=3" or using something like VSCode Go plugin, via "ui.codelenses" and "ui.diagnostic.annotations" configurations.
- johncolanduoni 1y ago
  I sometimes dream of a GCed language with a non-escaping pointer type. However to make it really useful (i.e. let you put it inside other non-escaping structs) you need something on the scale of the Rust borrow checker, which means adding a lot of complexity.
  - SkySkimmer 1y ago
    https://oxcaml.org/documentation/stack-allocation/intro/ https://oxcaml.org/documentation/stack-allocation/intro/ ?
  - pjmlp 1y ago
    Which is why several GC languages (the CS meaning of GC), are rather going down the path to keeping their approach to automatic resource management, plus type systems improvements for low level high performance code when needed. So you only go down into the complexity of affine types, linear types, effects, formal proofs, dependent types, if really needed, after spending time reasoning with a profiler. Now, this does not need to be so complex, since languages like Interlisp-D and Cedar at Xerox, that many GC languages have offered value types and explicit stack allocation. That alone is already good enough for most scenarios, provided people actually spend some time thinking about how to design their data structures instead of placing everything into the heap.
Thaxll 1y ago
pprof is pretty good, what do you need?
- defraudbah 1y ago
  yes, that's what I use, just wonder if there are alternatives. I am not sure how valgrind compares to it or the goref tool mentioned above, just asking around, does not hurt.
  - Thaxll 1y ago
    Alternative to solve what problem? pprof is very powerful, it's not missing much.
    - styluss 1y ago
      Pprof doesn't tell you if something was leaked aka still around. I fixed a leak recently because of misuse of a slice with code like slice = append(slice[1:], newElement) I only figured it out by looking at the pprof heap endpoint output and noticed there were multiple duplicate entries.
0x696C6961 1y ago
How are you getting "constant memory leaks" in a GC'd language?
- sim7c00 1y ago
  it's not hard. GC lets shit leak until it decided to clean it up... do you think they will enable Valgrind if there's no leaks?
  - chrsig 1y ago
    valgrind finds sooooo many more problems than just memory leaks uninitialized memory, illegal writes, etc... There's a lot of good stuff that could be discovered.
    - sim7c00 1y ago
      not to mention cachegrind, callgrind and other things it bundles. sorry, i guess when i say leaks i mean a bit more broad stuff :'). my own words are a bit leaky hah still doesnt mean i am wrong. GC doesnt clean up memory when its released but when it wants to, effectively offering opportunities to get that data after a program dont need it anymore. until some point in time u can usually not specify, just hint at. that in light of things like bad memory ordering between threads etc..can have nasty bugs... (raii has similar bugs but since its more determenistic you can program your way around lot of it more easily and reliably)
- johncolanduoni 1y ago
  Golang has a feature that I love in general but that makes it very easy to keep unintended allocations around. If you have a struct with a simple int field, and you store that somewhere as an *int, the entire struct and anything it points to will be kept alive. This is super useful for short-lived pointers, and super dangerous for long-lived pointers. Most other widely used GCed languages don’t allow the use of arbitrary interior pointers (though most GCs can actually handle them at the register level).
  - 1y ago
sethammons 1y ago
> constant memory leaks/pressure In Go, never launch a goroutine that you don't know exactly how it will be cleaned up.

looks very promising, one of the biggest issue in golang for me is profiling and constant memory leaks/pressure. Not sure if there is an alternative of what people use now

felixge 1y ago

I'd love to hear more! What kind of profiling issues are you running into? I'm assuming the inuse memory profiles are sometimes not good enough to track down leaks since they only show the allocation stack traces? Have you tried goref [1]?. What kind of memory pressure issues are you dealing with? [1] https://github.com/cloudwego/goref https://github.com/cloudwego/goref Disclaimer: I work on continuous profiling for Datadog and contribute to the profiling features in the runtime.

tsimionescu 1y ago
I for one am still mystified how it's possible that a GC language can't expose the GC roots in a memory profile. I've lost so many hours of my life manually trying to figure out what might be keeping some objects live, information the GC figures out every single time it runs...
- felixge 1y ago
  Do you think the GC roots alone (goroutine stacks with goroutine id, package globals) would be enough? I think in many cases you'd want the reference chains. The GC could certainly keep track of those, but at the expense of making things slower. My colleagues Nick and Daniel prototyped this at some point [1]. Alternatively the tracing of reference chains can be done on heap dumps, but it requires maintaining a partial replica of the GC in user space, see goref [2] for that approach. So it's not entirely trivial, but rest assured that it's definitely being considered by the Go project. You can see some discussions related to it here [3]. Disclaimer: I contribute to the Go runtime as part of my job at Datadog. I can't speak on behalf of the Go team. [1] https://go-review.googlesource.com/c/go/+/552736 https://go-review.googlesource.com/c/go/+/552736 [2] https://github.com/cloudwego/goref/blob/main/docs/principle.md https://github.com/cloudwego/goref/blob/main/docs/principle.... [3] https://github.com/golang/go/issues/57175 https://github.com/golang/go/issues/57175
defraudbah 1y ago
no, haven't heard of goref yet but will give it a shot! usually I go with pprof, like basic stuff and it helps. I would NOT say memory leak is the biggest or most common issue I see, however as time goes and services become more complicated what I often see in the metrics is how RAM gets eaten and does not get freed as time goes, so the app eats more and more memory as time goes and only restart helps. It's hard to call it memory leak in "original meaning of memory leak" but the memory does not get cleaned up because the choices I made and I want to understand how to make it better. Thanks for the tool!
- prerok 1y ago
  Sorry if this is a basic question but are you setting the GOMEMLIMIT? Also, are you running the code in a container? In K8s?

pjmlp 1y ago

Ideally, they would have learnt from other languages, and offered explicit control over what goes into the stack instead of relying into escape analysis alone. As it is, the only way to currently handle that is with " -gcflags -m=3" or using something like VSCode Go plugin, via "ui.codelenses" and "ui.diagnostic.annotations" configurations.

johncolanduoni 1y ago
I sometimes dream of a GCed language with a non-escaping pointer type. However to make it really useful (i.e. let you put it inside other non-escaping structs) you need something on the scale of the Rust borrow checker, which means adding a lot of complexity.
- SkySkimmer 1y ago
  https://oxcaml.org/documentation/stack-allocation/intro/ https://oxcaml.org/documentation/stack-allocation/intro/ ?
- pjmlp 1y ago
  Which is why several GC languages (the CS meaning of GC), are rather going down the path to keeping their approach to automatic resource management, plus type systems improvements for low level high performance code when needed. So you only go down into the complexity of affine types, linear types, effects, formal proofs, dependent types, if really needed, after spending time reasoning with a profiler. Now, this does not need to be so complex, since languages like Interlisp-D and Cedar at Xerox, that many GC languages have offered value types and explicit stack allocation. That alone is already good enough for most scenarios, provided people actually spend some time thinking about how to design their data structures instead of placing everything into the heap.

Thaxll 1y ago

pprof is pretty good, what do you need?

defraudbah 1y ago
yes, that's what I use, just wonder if there are alternatives. I am not sure how valgrind compares to it or the goref tool mentioned above, just asking around, does not hurt.
- Thaxll 1y ago
  Alternative to solve what problem? pprof is very powerful, it's not missing much.
  - styluss 1y ago
    Pprof doesn't tell you if something was leaked aka still around. I fixed a leak recently because of misuse of a slice with code like slice = append(slice[1:], newElement) I only figured it out by looking at the pprof heap endpoint output and noticed there were multiple duplicate entries.

0x696C6961 1y ago

How are you getting "constant memory leaks" in a GC'd language?

sim7c00 1y ago
it's not hard. GC lets shit leak until it decided to clean it up... do you think they will enable Valgrind if there's no leaks?
- chrsig 1y ago
  valgrind finds sooooo many more problems than just memory leaks uninitialized memory, illegal writes, etc... There's a lot of good stuff that could be discovered.
  - sim7c00 1y ago
    not to mention cachegrind, callgrind and other things it bundles. sorry, i guess when i say leaks i mean a bit more broad stuff :'). my own words are a bit leaky hah still doesnt mean i am wrong. GC doesnt clean up memory when its released but when it wants to, effectively offering opportunities to get that data after a program dont need it anymore. until some point in time u can usually not specify, just hint at. that in light of things like bad memory ordering between threads etc..can have nasty bugs... (raii has similar bugs but since its more determenistic you can program your way around lot of it more easily and reliably)
johncolanduoni 1y ago
Golang has a feature that I love in general but that makes it very easy to keep unintended allocations around. If you have a struct with a simple int field, and you store that somewhere as an *int, the entire struct and anything it points to will be kept alive. This is super useful for short-lived pointers, and super dangerous for long-lived pointers. Most other widely used GCed languages don’t allow the use of arbitrary interior pointers (though most GCs can actually handle them at the register level).
- 1y ago

sethammons 1y ago

> constant memory leaks/pressure In Go, never launch a goroutine that you don't know exactly how it will be cleaned up.