4 ms·

And I thought libc didn't use more than a few megs of ram. Here the difference seems to be like hundreds of megs.

by z92 8y ago

JulianMorrison 8y ago
Doesn't libc end up shared between all the things and amortized that way?
- de_Selby 8y ago
  Go programs are statically linked, you're referring to dynamic linking.
  - weberc2 8y ago
    Go doesn’t statically link against libc; maybe it’s possible with the likes of musl, but in general, libc is dynamically linked.
  - zbentley 8y ago
    > Go programs are statically linked Not necessarily true, unless you use strategies like the one suggested in the top level post above this. https://www.osso.nl/blog/golang-statically-linked/ https://www.osso.nl/blog/golang-statically-linked/ In practice, those dependencies are ubiquitous enough that the "DLL hell" half of the dynamic-linking problem is avoided. The other issues (system-dependent behavior quirks, estimation from static libraries, linker caching, etc. etc.) remain in many cases.
  - Arkanosis 8y ago
    Actually, the libc is dynamically linked here, as shown in the /proc/maps section of the article. So, to answer the GP: yes, resident memory from the libc ends up being amortized between all the processes that use it. But it doesn't even matter much as there's a mere 5 MiB of resident memory involved here. The “hundreds of MiB” are only of virtual memory, and that shouldn't be of any concern (see my other comment in this thread).
Arkanosis 8y ago
It's virtual memory, not physical memory. One should really see virtual memory as “addresses available for use within a single process”, not “space to store actual data, shared by all processes on a computer”. On modern 64 bits architectures, you have plenty of it to spend. To give an idea of how much of it you have: since Linux 2.6.11 on x86_64, one can address up to 128 TiB (that's 128 thousands of GiB) of virtual memory per process. Starting with Linux 4.12, one can address up to 128 PiB (that's 128 millions of GiB) of virtual memory per process. That's completely unrelated with how much physical memory you have.
- scott_s 8y ago
  Correct. When explaining it to students, I have used the metaphor of a housing developer building homes in a subdivision: going to the OS to "allocate" virtual memory is like getting a building permit from the government for a large swath of land. Having permission to build is not the same as a house actually being on that land (which is analogous to physical memory). Much as housing developers are incentivized to get permits for a large block of land at a time, general purpose memory allocators are incentivized to ask the OS for permission to use large blocks of virtual memory at a time. Asking the OS for virtual memory is very slow, so to avoid doing it a lot, general purpose memory allocators ask for big chunks and manage those chunks themselves.

And I thought libc didn't use more than a few megs of ram. Here the difference seems to be like hundreds of megs.

by z92 8y ago

And I thought libc didn't use more than a few megs of ram. Here the difference seems to be like hundreds of megs.

JulianMorrison 8y ago

Doesn't libc end up shared between all the things and amortized that way?

de_Selby 8y ago
Go programs are statically linked, you're referring to dynamic linking.
- weberc2 8y ago
  Go doesn’t statically link against libc; maybe it’s possible with the likes of musl, but in general, libc is dynamically linked.
- zbentley 8y ago
  > Go programs are statically linked Not necessarily true, unless you use strategies like the one suggested in the top level post above this. https://www.osso.nl/blog/golang-statically-linked/ https://www.osso.nl/blog/golang-statically-linked/ In practice, those dependencies are ubiquitous enough that the "DLL hell" half of the dynamic-linking problem is avoided. The other issues (system-dependent behavior quirks, estimation from static libraries, linker caching, etc. etc.) remain in many cases.
- Arkanosis 8y ago
  Actually, the libc is dynamically linked here, as shown in the /proc/maps section of the article. So, to answer the GP: yes, resident memory from the libc ends up being amortized between all the processes that use it. But it doesn't even matter much as there's a mere 5 MiB of resident memory involved here. The “hundreds of MiB” are only of virtual memory, and that shouldn't be of any concern (see my other comment in this thread).

Arkanosis 8y ago

It's virtual memory, not physical memory. One should really see virtual memory as “addresses available for use within a single process”, not “space to store actual data, shared by all processes on a computer”. On modern 64 bits architectures, you have plenty of it to spend. To give an idea of how much of it you have: since Linux 2.6.11 on x86_64, one can address up to 128 TiB (that's 128 thousands of GiB) of virtual memory per process. Starting with Linux 4.12, one can address up to 128 PiB (that's 128 millions of GiB) of virtual memory per process. That's completely unrelated with how much physical memory you have.

scott_s 8y ago
Correct. When explaining it to students, I have used the metaphor of a housing developer building homes in a subdivision: going to the OS to "allocate" virtual memory is like getting a building permit from the government for a large swath of land. Having permission to build is not the same as a house actually being on that land (which is analogous to physical memory). Much as housing developers are incentivized to get permits for a large block of land at a time, general purpose memory allocators are incentivized to ask the OS for permission to use large blocks of virtual memory at a time. Asking the OS for virtual memory is very slow, so to avoid doing it a lot, general purpose memory allocators ask for big chunks and manage those chunks themselves.