8 ms·

Programs compiled by Go 1.11 allocate an unreasonable amount of virtual memory

137 points by networkimprov 8y ago

weberc2 8y ago
Impact: > The significant increase in virtual memory usage is usually not an issue, however security sensitive programs often lock their memory, causing a far greater performance degradation on low-spec computer hosts.
- exacube 8y ago
  I'd love some further explanation here: what does locking memory mean in this context? For what purpose do security sensitive programs lock their memory? what is the performance degradation that happens with low-spec computers?
  - SteveNuts 8y ago
    It tells the OS to never swap memory allocated to the process to disk. You should always lock memory if you're going to be storing crypto keys, etc. since once the pages are swapped to disk you're vulnerable to someone pulling the swap partition out and reading it.
    - rkeene2 8y ago
      It is significant to note that: 1. You don't have to lock all your memory (although it may be hard to capture all the intermediate buffers if you don't). 2. You still need to clear the memory buffers after they're no longer going to be used, otherwise other processes can read /proc/kcore, etc (or cool your RAM and extract it and put it another system) 3. It is possible to encrypt the swap partition with a randomly generated key at boot
  - richard_todd 8y ago
    One locks memory to keep it from being swapped to disk. You can imagine if you have sensitive data, you want to keep it as ephemeral as possible. Do even security-focused go programs typically lock _all_ of their virtual memory, or just the sensitive pages? I don’t know. But if a bunch of programs hogged hundreds of megabytes of unused RAM each, that’s the problem alluded to on low resource systems.
  - deathanatos 8y ago
    > what does locking memory mean in this context? I think the person is referring to the mlock() and mlockall() functions (or equivalents on other OS), which keep pages resident / prevents pages from being paged out. Forces them to remain in RAM. It can be used to, e.g., prevent an encryption key or password from being swapped out to disk, where it might then be recoverable. (Personally, this is why I encrypt swap.) > what is the performance degradation that happens with low-spec computers? Locking a larger portion of RAM means less room for the OS to page out unused pages and free up the space for other programs. While one can try to selectively lock buffers with sensitive data with mlock(), you have to be sure they aren't copied into other buffers that aren't locked (and could thus be subsequently paged out). If you're writing a UI program that displays or receives those in a widget, this might be harder (you might not have access to the internal buffer of the widget, as it is an "implementation detail" of your library), and locking the entire process might be a simpler solution (albeit being a bigger hammer).
    - rkeene2 8y ago
      Linux cgroups and Solaris containers atleast also provide a way to avoid paging to disk, without modifying the program.
    - scott_s 8y ago
      Good explanations of what and why one might lock memory. But, in the context of this overall discussion, I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory. So a process allocating a lot of virtual memory up front should not impact other processes which have locked some of their memory into physical memory.
      - deathanatos 8y ago
        > I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory. It will, if you mlock() it, I believe. The manual page notes "real-time processes" as a main user of mlock() (the other being the cryptographic uses I hinted at); it cites their use case as locking the page to avoid delays due to paging during critical sections. In order for that to work, the OS would need to bring the pages in, at the time of locking; so at that point, a large virtual allocation becomes equivalent to a physical one.
- rkeene2 8y ago
  I assume this is referring to some mechanism like `mlockall(2)` or using cgroups to disable using paging. Is the "excess" memory dirtied ? It's not clear why having the pages mapped but unaccessed entirely would cause a performance issue.
  - irishsultan 8y ago
    From the manpage of mlockall > All pages that contain a part of the specified address range are guaranteed to be resident in RAM when the call returns successfully; the pages are guaranteed to stay in RAM until later unlocked. So it doesn't matter if the memory is dirty, as long as it's marked as locked.
    - rkeene2 8y ago
      But since the memory is never actually backed by anything, it doesn't have a real effect except in the accounting of the process. No additional memory is used by locking it.
      - wolfgang42 8y ago
        Doesn't "guaranteed to be resident in RAM" mean the memory gets backing?
        rkeene2 8y ago
        No, because the pages aren't mapped to anything -- not disk, RAM, or an external paging device. Allocating a chunk of memory typically doesn't map those pages to anything immediately (since there's no benefit to doing that extra work). The current Linux man page gives a bit more insight: mlockall() and munlockall() mlockall() locks all pages mapped into the address space of the calling process. This includes the pages of the code, data and stack segment, as well as shared libraries, user space kernel data, shared memory, and memory-mapped files. All mapped pages are guaranteed to be resident in RAM when the call returns successfully; the pages are guaranteed to stay in RAM until later unlocked. The flags argument is con‐ structed as the bitwise OR of one or more of the following constants: MCL_CURRENT Lock all pages which are currently mapped into the address space of the process. MCL_FUTURE Lock all pages which will become mapped into the address space of the process in the future. These could be, for instance, new pages required by a grow‐ ing heap and stack as well as new mem‐ ory-mapped files or shared memory regions.
- maximilianburke 8y ago
  Is this really a problem? Operating systems usually cap the amount of locked pages to prevent the system from being DoS'd; on Linux it can be quite low (16kb). In all the code I've written using locked memory, these limitations have forced me to use separate arenas for the locked/sensitive memory because of its scarcity. In general, it would be incompatible with Go's garbage collected heap unless the GC heap has the concept of "sensitive" objects and pools them in locked memory (which, AFAICT, it doesn't); or the heap limited itself to the locked memory limit (impractical) If you allocate and maintain unmanaged locked memory yourself in Go it shouldn't matter if the 1.11 runtime uses more virtual memory since you've separated yourself from the problem by going your own route.
CJefferson 8y ago
I do this in some programs, it makes many pieces of coding much easier if I know I will never have to move some objects as they grow. The ideal for me would be a function that marked some memory as "these addresses are taken, do not give them out to malloc, or anything else", but which still required me to actually "ask" for the memory before using it, so it didn't look like I was using 64gb of memory at startup. Is that possible?
- sjmulder 8y ago
  On Windows that's VirtualAlloc with MEM_RESERVE: Reserves a range of the process's virtual address space without allocating any actual physical storage in memory or in the paging file on disk. https://msdn.microsoft.com/en-us/library/Aa366887(v=VS.85).aspx https://msdn.microsoft.com/en-us/library/Aa366887(v=VS.85).a... Perhaps mmap() can achieve the same on Unices?
  - cyphar 8y ago
    This is what Linux does by default for anonymous memory (it's called overcommitment). Go makes very liberal use of this, but it has other issues (namely Linux will kill a process if it goes over a certain overcommitment threshold).
    - scott_s 8y ago
      I'm not sure overcommitment is the same thing: https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou... Reading that, I think overcommitment is to determine the kernel's behavior when you try to allocate more virtual memory than physical memory that is present on the system. That is a different (but related) concern from the fact that mmap() will allocate virtual memory but not physical memory. (That is, mmap() reserves locations in the address space, but you don't have any physical memory backing it until you use that memory.)
      - cyphar 8y ago
ahh 8y ago
Unreasonable according to you. :) The nice thing about virtual memory is that it's, well, virtual. It costs you almost nothing until you've touched it. (Fun exercise for the reader: measure the kernel overhead for an unused 1 TiB VMA.) But creating huge spaces--that terabyte mmap wasn't theoretical--that stay untouched is hugely algorithmically useful, especially for things like malloc implementations. Why does it bother people? Two reasons. First is mlock to avoid swap. This is solvable in much better ways--I'm a fan of disabling swap in many cases anyway. Second is that, absent cgroups, it's difficult to put hard limits on memory usage in Linux. So people, looking under the streetlight, put limits on virtual usage, even though that's not what they care about limiting! Then they get angry when you break it. My refrain here, as in many cases (see for example measuring process CPU time spent in kernel mode): "X is impossible" doesn't justify Y unless Y correctly solves the problem X does. (I spent years in charge of a major memory allocator so this is a battle I've fought too many times.)
- exabrial 8y ago
  You cannot possibly be serious: package main func main() { for { } } Using 100mb+ of memory?
  - outside1234 8y ago
    It is not "using" but allocating the addresses. The committed memory will be vastly smaller. This is how virtual memory works.
  - weberc2 8y ago
    To quote the parent: `The nice thing about virtual memory is that it's, well, virtual. It costs you almost nothing until you've touched it.`
  - Thaxll 8y ago
    Virtual memory is different than resident / RSS ...
  - masklinn 8y ago
    > Using 100mb+ of memory? It's not using them. On modern OSX, processes get something like 2.4GB vmem by default, even if they do nothing. #include <unistd.h> int main() { for(;;) { sleep(10); } } is reported as 2377M VMEM by top/htop, on 10.11.
kccqzy 8y ago
Check out GHC 8.0+ where the same program will allocate a terabyte of virtual memory ;) That said, I don't find it unreasonable at all. Just reserving some bits in the address space isn't unreasonable. It makes the real allocation code simpler.
- xfalcox 8y ago
  Yes, we are running a Haskell based HTTPS redirector and people noticed this.

137 points by networkimprov 8y ago

Impact: > The significant increase in virtual memory usage is usually not an issue, however security sensitive programs often lock their memory, causing a far greater performance degradation on low-spec computer hosts.

exacube 8y ago
I'd love some further explanation here: what does locking memory mean in this context? For what purpose do security sensitive programs lock their memory? what is the performance degradation that happens with low-spec computers?
- SteveNuts 8y ago
  It tells the OS to never swap memory allocated to the process to disk. You should always lock memory if you're going to be storing crypto keys, etc. since once the pages are swapped to disk you're vulnerable to someone pulling the swap partition out and reading it.
  - rkeene2 8y ago
    It is significant to note that: 1. You don't have to lock all your memory (although it may be hard to capture all the intermediate buffers if you don't). 2. You still need to clear the memory buffers after they're no longer going to be used, otherwise other processes can read /proc/kcore, etc (or cool your RAM and extract it and put it another system) 3. It is possible to encrypt the swap partition with a randomly generated key at boot
- richard_todd 8y ago
  One locks memory to keep it from being swapped to disk. You can imagine if you have sensitive data, you want to keep it as ephemeral as possible. Do even security-focused go programs typically lock _all_ of their virtual memory, or just the sensitive pages? I don’t know. But if a bunch of programs hogged hundreds of megabytes of unused RAM each, that’s the problem alluded to on low resource systems.
- deathanatos 8y ago
  > what does locking memory mean in this context? I think the person is referring to the mlock() and mlockall() functions (or equivalents on other OS), which keep pages resident / prevents pages from being paged out. Forces them to remain in RAM. It can be used to, e.g., prevent an encryption key or password from being swapped out to disk, where it might then be recoverable. (Personally, this is why I encrypt swap.) > what is the performance degradation that happens with low-spec computers? Locking a larger portion of RAM means less room for the OS to page out unused pages and free up the space for other programs. While one can try to selectively lock buffers with sensitive data with mlock(), you have to be sure they aren't copied into other buffers that aren't locked (and could thus be subsequently paged out). If you're writing a UI program that displays or receives those in a widget, this might be harder (you might not have access to the internal buffer of the widget, as it is an "implementation detail" of your library), and locking the entire process might be a simpler solution (albeit being a bigger hammer).
  - rkeene2 8y ago
    Linux cgroups and Solaris containers atleast also provide a way to avoid paging to disk, without modifying the program.
  - scott_s 8y ago
    Good explanations of what and why one might lock memory. But, in the context of this overall discussion, I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory. So a process allocating a lot of virtual memory up front should not impact other processes which have locked some of their memory into physical memory.
    - deathanatos 8y ago
      > I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory. It will, if you mlock() it, I believe. The manual page notes "real-time processes" as a main user of mlock() (the other being the cryptographic uses I hinted at); it cites their use case as locking the page to avoid delays due to paging during critical sections. In order for that to work, the OS would need to bring the pages in, at the time of locking; so at that point, a large virtual allocation becomes equivalent to a physical one.
rkeene2 8y ago
I assume this is referring to some mechanism like `mlockall(2)` or using cgroups to disable using paging. Is the "excess" memory dirtied ? It's not clear why having the pages mapped but unaccessed entirely would cause a performance issue.
- irishsultan 8y ago
  From the manpage of mlockall > All pages that contain a part of the specified address range are guaranteed to be resident in RAM when the call returns successfully; the pages are guaranteed to stay in RAM until later unlocked. So it doesn't matter if the memory is dirty, as long as it's marked as locked.
  - rkeene2 8y ago
    But since the memory is never actually backed by anything, it doesn't have a real effect except in the accounting of the process. No additional memory is used by locking it.
    - wolfgang42 8y ago
      Doesn't "guaranteed to be resident in RAM" mean the memory gets backing?
      - rkeene2 8y ago
        No, because the pages aren't mapped to anything -- not disk, RAM, or an external paging device. Allocating a chunk of memory typically doesn't map those pages to anything immediately (since there's no benefit to doing that extra work). The current Linux man page gives a bit more insight: mlockall() and munlockall() mlockall() locks all pages mapped into the address space of the calling process. This includes the pages of the code, data and stack segment, as well as shared libraries, user space kernel data, shared memory, and memory-mapped files. All mapped pages are guaranteed to be resident in RAM when the call returns successfully; the pages are guaranteed to stay in RAM until later unlocked. The flags argument is con‐ structed as the bitwise OR of one or more of the following constants: MCL_CURRENT Lock all pages which are currently mapped into the address space of the process. MCL_FUTURE Lock all pages which will become mapped into the address space of the process in the future. These could be, for instance, new pages required by a grow‐ ing heap and stack as well as new mem‐ ory-mapped files or shared memory regions.
maximilianburke 8y ago
Is this really a problem? Operating systems usually cap the amount of locked pages to prevent the system from being DoS'd; on Linux it can be quite low (16kb). In all the code I've written using locked memory, these limitations have forced me to use separate arenas for the locked/sensitive memory because of its scarcity. In general, it would be incompatible with Go's garbage collected heap unless the GC heap has the concept of "sensitive" objects and pools them in locked memory (which, AFAICT, it doesn't); or the heap limited itself to the locked memory limit (impractical) If you allocate and maintain unmanaged locked memory yourself in Go it shouldn't matter if the 1.11 runtime uses more virtual memory since you've separated yourself from the problem by going your own route.

CJefferson 8y ago

I do this in some programs, it makes many pieces of coding much easier if I know I will never have to move some objects as they grow. The ideal for me would be a function that marked some memory as "these addresses are taken, do not give them out to malloc, or anything else", but which still required me to actually "ask" for the memory before using it, so it didn't look like I was using 64gb of memory at startup. Is that possible?

sjmulder 8y ago
On Windows that's VirtualAlloc with MEM_RESERVE: Reserves a range of the process's virtual address space without allocating any actual physical storage in memory or in the paging file on disk. https://msdn.microsoft.com/en-us/library/Aa366887(v=VS.85).aspx https://msdn.microsoft.com/en-us/library/Aa366887(v=VS.85).a... Perhaps mmap() can achieve the same on Unices?
- cyphar 8y ago
  This is what Linux does by default for anonymous memory (it's called overcommitment). Go makes very liberal use of this, but it has other issues (namely Linux will kill a process if it goes over a certain overcommitment threshold).
  - scott_s 8y ago
    I'm not sure overcommitment is the same thing: https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou... Reading that, I think overcommitment is to determine the kernel's behavior when you try to allocate more virtual memory than physical memory that is present on the system. That is a different (but related) concern from the fact that mmap() will allocate virtual memory but not physical memory. (That is, mmap() reserves locations in the address space, but you don't have any physical memory backing it until you use that memory.)
    - cyphar 8y ago

ahh 8y ago

Unreasonable according to you. :) The nice thing about virtual memory is that it's, well, virtual. It costs you almost nothing until you've touched it. (Fun exercise for the reader: measure the kernel overhead for an unused 1 TiB VMA.) But creating huge spaces--that terabyte mmap wasn't theoretical--that stay untouched is hugely algorithmically useful, especially for things like malloc implementations. Why does it bother people? Two reasons. First is mlock to avoid swap. This is solvable in much better ways--I'm a fan of disabling swap in many cases anyway. Second is that, absent cgroups, it's difficult to put hard limits on memory usage in Linux. So people, looking under the streetlight, put limits on virtual usage, even though that's not what they care about limiting! Then they get angry when you break it. My refrain here, as in many cases (see for example measuring process CPU time spent in kernel mode): "X is impossible" doesn't justify Y unless Y correctly solves the problem X does. (I spent years in charge of a major memory allocator so this is a battle I've fought too many times.)

exabrial 8y ago
You cannot possibly be serious: package main func main() { for { } } Using 100mb+ of memory?
- outside1234 8y ago
  It is not "using" but allocating the addresses. The committed memory will be vastly smaller. This is how virtual memory works.
- weberc2 8y ago
  To quote the parent: `The nice thing about virtual memory is that it's, well, virtual. It costs you almost nothing until you've touched it.`
- Thaxll 8y ago
  Virtual memory is different than resident / RSS ...
- masklinn 8y ago
  > Using 100mb+ of memory? It's not using them. On modern OSX, processes get something like 2.4GB vmem by default, even if they do nothing. #include <unistd.h> int main() { for(;;) { sleep(10); } } is reported as 2377M VMEM by top/htop, on 10.11.

kccqzy 8y ago

Check out GHC 8.0+ where the same program will allocate a terabyte of virtual memory ;) That said, I don't find it unreasonable at all. Just reserving some bits in the address space isn't unreasonable. It makes the real allocation code simpler.

xfalcox 8y ago
Yes, we are running a Haskell based HTTPS redirector and people noticed this.