7 ms·
A deep dive into the Go memory allocator and garbage collector
- beliu 8y agoGopherCon liveblogger here. The room was jampacked for this talk and it did not disappoint. A very detailed tour of how memory allocation and GC works in Go. I learned a ton. Eben (https://twitter.com/_emfree_ https://twitter.com/_emfree_) has also posted his slides here: https://speakerdeck.com/emfree/allocator-wrestling https://speakerdeck.com/emfree/allocator-wrestling. FYI, this was part of a technical liveblog that covered all the talks at GopherCon this year in a similar fashion. If you missed the conference or are looking to review material, you might find https://sourcegraph.com/gophercon https://sourcegraph.com/gophercon helpful. (Disclosure: I'm CTO of Sourcegraph—we organized the liveblog and love Go!)
- ebikelaw 8y agoTheir allocator architecture seems very similar to tcmalloc, which maybe makes sense if all those people sit together at Google.
- dullgiulio 8y agoYes, it was an explicit adaptation, as documented in the sources themselves.
- weberc2 8y agoI wonder if they considered jemalloc? Apparently tcmalloc does suffer from fragmentation issues over time, and I believe I've seen related issues crop up on the Go issue tracker. https://www.quora.com/Is-tcmalloc-stable-enough-for-production-use https://www.quora.com/Is-tcmalloc-stable-enough-for-producti... EDIT: Not sure why I'm being downvoted; if this is a contentious topic, I wasn't aware. I'm certainly not trying to make a point one way or another; I only posted this hoping someone who was more knowledgeable than me could respond.
- joaodlf 8y agoIf this topic interests you, André Carvalho did another great talk on the subject at Gophercon UK earlier this month: https://speakerdeck.com/andrestc/understanding-go-memory-allocation-gophercon-uk https://speakerdeck.com/andrestc/understanding-go-memory-all...
- sephamorr 8y agoThe suggestion to reuse objects, rather than reallocate temporaries (e.g. inside a loop body) was intriguing. Coming from C/C++ where stack allocations are approximately 'free', I tend to scope stack variables as narrowly as possible for readability and to help the compiler break dependencies. This is an interesting paradigm change that I wouldn't have expected from Go.
- mwuertinger 8y agoAs far as I understand this Go will always try to keep objects on the stack because it is practically free (just as in C/C++). However, if the object escapes to the heap then allocation gets expensive (just as in any other language) and it can be beneficial to reuse objects instead of allocating new ones for every iteration.
- HohPum1l 8y agoIdeally it would still stack-allocate where beneficial and only materialize the object once it escapes to the heap, at least that's what graalvm does
- presscast 8y ago>However, if the object escapes to the heap Can this be detected statically?
- mwuertinger 8y agoYes this is decided at compile time. If you build your program with `-gcflags '-m'` the compiler will print out detailed escape analysis information. Here's more information: http://www.agardner.me/golang/garbage/collection/gc/escape/analysis/2015/10/18/go-escape-analysis.html http://www.agardner.me/golang/garbage/collection/gc/escape/a...
- presscast 8y agoNice! So this could presumably be built into a linter (or similar) and be flagged in the editor ... interesting...
- amelius 8y agoI wish a powerful GC like the one in GoLang was built into e.g. LLVM, so other languages could profit from it as well.
- masklinn 8y ago1. LLVM has at several built-in GC strategies and hooks to BYO: https://llvm.org/docs/GarbageCollection.html#built-in-gc-strategies https://llvm.org/docs/GarbageCollection.html#built-in-gc-str... 2. Go does not have "a powerful GC", it has a GC built quite specifically for it and the applications it's intended to support with low-latency, middling throughput, and the assumption that allocations are long-lived (because short-lived objects don't escape and thus don't get heap-allocated) 3. Which is the issue of providing generic GC strategies: the type of GC you want and how you want to tune it is very much a factor of the language's semantics and its intended use cases, hence the official JVM shipping with something like half a dozen different GCs, and third-party JVMs having their own with their own tradeoffs
- snaky 8y ago> the type of GC you want and how you want to tune it is very much a factor of the language's semantics and its intended use cases And when one have some (parts, modules, libraries) in your program which are best suited with different GC behaviour, things tend to become complicated. And cross-language boundaries is yet another talk.
- amelius 8y ago1. I'm aware of this, but it seems LLVM's built-in GC is not a concurrent collector, which is quite a big deal. 2. The assumption that allocations are long-lived can often be made because a compiler can perform escape-analysis (and in fact LLVM could provide utilities for this). 3. Yes, I'm not saying that GoLang's GC (or an adaptation thereof) is the holy grail, just that it would be very useful for many other languages. And work in this area could help bring LLVM's suite of GCs to a competitive, world-class level.
- M_Bakhtiari 8y agoI believe Ravenbrook MPS is designed to be pluggable in such a way.