2 ms·

I wish Go recorded the timestamp of goroutine and let you access them. An app I work on recently had a bug where goroutines would slowly build up over time. Tu

I wish Go recorded the timestamp of goroutine and let you access them. An app I work on recently had a bug where goroutines would slowly build up over time. Turns out the bug is in the Growthbook SDK [1]. We can monitor the number of goroutines, but having a large number of goroutines waiting in the location that gets stuck is normal — we can only see such a problem over multiple days, in that the minimum value slowly goes up. If Go could tell you the timestamp of the oldest goroutines as part of the pprof dump, we could have an alert, and it would work for any such leak. [1] https://github.com/growthbook/growthbook-golang/pull/28 https://github.com/growthbook/growthbook-golang/pull/28

jmholla 3y ago
Language support would be great, but could you add logs that record the creating and destruction of goroutines, giving them a unique UUID so you can track which one's haven't exited? Edit: Also, maybe the tool at this comment could've helped you? https://news.ycombinator.com/item?id=39817775 https://news.ycombinator.com/item?id=39817775
- atombender 3y ago
  How would one log that? In our case the bug was in a goroutine started by a third-party library, so we don't have control over when it starts or exits. I don't think Goleak would have helped here, because I believe it doesn't support concurrency. It's really designed to run in tests, not in production. It parses and searches stack traces, so it's not going to be performant.
  - physicles 3y ago
    I guess you’d need to fork the third-party library (or, if you can build locally, just yolo and chmod +w) and inject something to track those goroutine metrics. But I suppose by the time you’re doing that, you already know where the problem is.
prosunpraiser 3y ago
Execution traces have a goroutine profile which outputs the count of goroutines as well. That can be used for an alert as well - though it would require parsing the trace output. They recently made some changes to give a structured API over trace data - maybe use that?

jmholla 3y ago

Language support would be great, but could you add logs that record the creating and destruction of goroutines, giving them a unique UUID so you can track which one's haven't exited? Edit: Also, maybe the tool at this comment could've helped you? https://news.ycombinator.com/item?id=39817775 https://news.ycombinator.com/item?id=39817775

atombender 3y ago
How would one log that? In our case the bug was in a goroutine started by a third-party library, so we don't have control over when it starts or exits. I don't think Goleak would have helped here, because I believe it doesn't support concurrency. It's really designed to run in tests, not in production. It parses and searches stack traces, so it's not going to be performant.
- physicles 3y ago
  I guess you’d need to fork the third-party library (or, if you can build locally, just yolo and chmod +w) and inject something to track those goroutine metrics. But I suppose by the time you’re doing that, you already know where the problem is.

prosunpraiser 3y ago

Execution traces have a goroutine profile which outputs the count of goroutines as well. That can be used for an alert as well - though it would require parsing the trace output. They recently made some changes to give a structured API over trace data - maybe use that?