4 ms·

Go Memory Management

165 points by ngaut 8y ago

zahreeley 8y ago
Very good explanation of memory management
TheDong 8y ago
Spoilers for why the simpler program is different: It doesn't do networking, which means it doesn't use any cgo. The networking stack by default links against libc, but if you don't use it, your go program doesn't have to link against libc. You can always configure the CGO_ENABLED or -tags netgo flag in order to cause networking to not use libc either.
- z92 8y ago
  And I thought libc didn't use more than a few megs of ram. Here the difference seems to be like hundreds of megs.
  - JulianMorrison 8y ago
    Doesn't libc end up shared between all the things and amortized that way?
    - de_Selby 8y ago
      Go programs are statically linked, you're referring to dynamic linking.
      - weberc2 8y ago
        Go doesn’t statically link against libc; maybe it’s possible with the likes of musl, but in general, libc is dynamically linked.
      - zbentley 8y ago
        > Go programs are statically linked Not necessarily true, unless you use strategies like the one suggested in the top level post above this. https://www.osso.nl/blog/golang-statically-linked/ https://www.osso.nl/blog/golang-statically-linked/ In practice, those dependencies are ubiquitous enough that the "DLL hell" half of the dynamic-linking problem is avoided. The other issues (system-dependent behavior quirks, estimation from static libraries, linker caching, etc. etc.) remain in many cases.
      - Arkanosis 8y ago
        Actually, the libc is dynamically linked here, as shown in the /proc/maps section of the article. So, to answer the GP: yes, resident memory from the libc ends up being amortized between all the processes that use it. But it doesn't even matter much as there's a mere 5 MiB of resident memory involved here. The “hundreds of MiB” are only of virtual memory, and that shouldn't be of any concern (see my other comment in this thread).
  - Arkanosis 8y ago
    It's virtual memory, not physical memory. One should really see virtual memory as “addresses available for use within a single process”, not “space to store actual data, shared by all processes on a computer”. On modern 64 bits architectures, you have plenty of it to spend. To give an idea of how much of it you have: since Linux 2.6.11 on x86_64, one can address up to 128 TiB (that's 128 thousands of GiB) of virtual memory per process. Starting with Linux 4.12, one can address up to 128 PiB (that's 128 millions of GiB) of virtual memory per process. That's completely unrelated with how much physical memory you have.
    - scott_s 8y ago
      Correct. When explaining it to students, I have used the metaphor of a housing developer building homes in a subdivision: going to the OS to "allocate" virtual memory is like getting a building permit from the government for a large swath of land. Having permission to build is not the same as a house actually being on that land (which is analogous to physical memory). Much as housing developers are incentivized to get permits for a large block of land at a time, general purpose memory allocators are incentivized to ask the OS for permission to use large blocks of virtual memory at a time. Asking the OS for virtual memory is very slow, so to avoid doing it a lot, general purpose memory allocators ask for big chunks and manage those chunks themselves.
- presscast 8y ago
  What's the trade-off when disabling CGO?
- otterley 8y ago
  Networking doesn’t require cgo. Only programs that expliticly call C functions (i.e. those that use ‘import “C”’) do.
  - TheDong 8y ago
    It doesn't require cgo, but it defaults to it on linux. To quote from the official docs: https://golang.org/pkg/net/#hdr-Name_Resolution https://golang.org/pkg/net/#hdr-Name_Resolution > When cgo is available, the cgo-based resolver is used instead under a variety of conditions: .... when /etc/resolv.conf or /etc/nsswitch.conf specify the use of features that the Go resolver does not implement, and when the name being looked up ends in .local or is an mDNS name. Almost any linux machine's `/etc/nsswitch.conf` will contain features that the go resolver doesn't implement. Because of that, almost any dns resolution will default to cgo. In addition, selecting between netgo and cgo is a runtime feature by default (see GODEBUG=netdns=go|c), so it still has to build it in regardless unless it's explicitly disabled. You can also determine this by using `ldd` on a go binary. > Only programs that expliticly call C functions (i.e. those that use ‘import “C”’) do. That is exactly what go uses in programs that use networking. Here's the 'import "C"' in question: https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417afcaab722b0727c/src/net/cgo_linux.go#L9-L12 https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417a...
    - otterley 8y ago
      Thanks for the clarification!
    - TheDong 8y ago
      (note for anyone coming by in the future: that link is not the best one, https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417afcaab722b0727c/src/net/cgo_unix.go#L5-L18 https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417a... is where most of the cgo code for linux is)
    - e12e 8y ago
      > Almost any linux machine's `/etc/nsswitch.conf` will contain features that the go resolver doesn't implement. Do you know of a short list of top features missing? I would kinda hope that modern and generally useful features would be in netgo. And old rotted code, security risks and anachronisms would be abandoned. But from your tone, I gather that this is more than a "people only use 10%, but never the same 10%" type of thing?
deleted 8y ago
[deleted]

165 points by ngaut 8y ago

Very good explanation of memory management

Spoilers for why the simpler program is different: It doesn't do networking, which means it doesn't use any cgo. The networking stack by default links against libc, but if you don't use it, your go program doesn't have to link against libc. You can always configure the CGO_ENABLED or -tags netgo flag in order to cause networking to not use libc either.

z92 8y ago
And I thought libc didn't use more than a few megs of ram. Here the difference seems to be like hundreds of megs.
- JulianMorrison 8y ago
  Doesn't libc end up shared between all the things and amortized that way?
  - de_Selby 8y ago
    Go programs are statically linked, you're referring to dynamic linking.
    - weberc2 8y ago
      Go doesn’t statically link against libc; maybe it’s possible with the likes of musl, but in general, libc is dynamically linked.
    - zbentley 8y ago
      > Go programs are statically linked Not necessarily true, unless you use strategies like the one suggested in the top level post above this. https://www.osso.nl/blog/golang-statically-linked/ https://www.osso.nl/blog/golang-statically-linked/ In practice, those dependencies are ubiquitous enough that the "DLL hell" half of the dynamic-linking problem is avoided. The other issues (system-dependent behavior quirks, estimation from static libraries, linker caching, etc. etc.) remain in many cases.
    - Arkanosis 8y ago
      Actually, the libc is dynamically linked here, as shown in the /proc/maps section of the article. So, to answer the GP: yes, resident memory from the libc ends up being amortized between all the processes that use it. But it doesn't even matter much as there's a mere 5 MiB of resident memory involved here. The “hundreds of MiB” are only of virtual memory, and that shouldn't be of any concern (see my other comment in this thread).
- Arkanosis 8y ago
  It's virtual memory, not physical memory. One should really see virtual memory as “addresses available for use within a single process”, not “space to store actual data, shared by all processes on a computer”. On modern 64 bits architectures, you have plenty of it to spend. To give an idea of how much of it you have: since Linux 2.6.11 on x86_64, one can address up to 128 TiB (that's 128 thousands of GiB) of virtual memory per process. Starting with Linux 4.12, one can address up to 128 PiB (that's 128 millions of GiB) of virtual memory per process. That's completely unrelated with how much physical memory you have.
  - scott_s 8y ago
    Correct. When explaining it to students, I have used the metaphor of a housing developer building homes in a subdivision: going to the OS to "allocate" virtual memory is like getting a building permit from the government for a large swath of land. Having permission to build is not the same as a house actually being on that land (which is analogous to physical memory). Much as housing developers are incentivized to get permits for a large block of land at a time, general purpose memory allocators are incentivized to ask the OS for permission to use large blocks of virtual memory at a time. Asking the OS for virtual memory is very slow, so to avoid doing it a lot, general purpose memory allocators ask for big chunks and manage those chunks themselves.
presscast 8y ago
What's the trade-off when disabling CGO?
otterley 8y ago
Networking doesn’t require cgo. Only programs that expliticly call C functions (i.e. those that use ‘import “C”’) do.
- TheDong 8y ago
  It doesn't require cgo, but it defaults to it on linux. To quote from the official docs: https://golang.org/pkg/net/#hdr-Name_Resolution https://golang.org/pkg/net/#hdr-Name_Resolution > When cgo is available, the cgo-based resolver is used instead under a variety of conditions: .... when /etc/resolv.conf or /etc/nsswitch.conf specify the use of features that the Go resolver does not implement, and when the name being looked up ends in .local or is an mDNS name. Almost any linux machine's `/etc/nsswitch.conf` will contain features that the go resolver doesn't implement. Because of that, almost any dns resolution will default to cgo. In addition, selecting between netgo and cgo is a runtime feature by default (see GODEBUG=netdns=go|c), so it still has to build it in regardless unless it's explicitly disabled. You can also determine this by using `ldd` on a go binary. > Only programs that expliticly call C functions (i.e. those that use ‘import “C”’) do. That is exactly what go uses in programs that use networking. Here's the 'import "C"' in question: https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417afcaab722b0727c/src/net/cgo_linux.go#L9-L12 https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417a...
  - otterley 8y ago
    Thanks for the clarification!
  - TheDong 8y ago
    (note for anyone coming by in the future: that link is not the best one, https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417afcaab722b0727c/src/net/cgo_unix.go#L5-L18 https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417a... is where most of the cgo code for linux is)
  - e12e 8y ago
    > Almost any linux machine's `/etc/nsswitch.conf` will contain features that the go resolver doesn't implement. Do you know of a short list of top features missing? I would kinda hope that modern and generally useful features would be in netgo. And old rotted code, security risks and anachronisms would be abandoned. But from your tone, I gather that this is more than a "people only use 10%, but never the same 10%" type of thing?

deleted 8y ago

[deleted]