9 ms·

Unreasonable according to you. :) The nice thing about virtual memory is that it's, well, virtual. It costs you almost nothing until you've touched it. (Fun ex

by ahh 8y ago

Unreasonable according to you. :) The nice thing about virtual memory is that it's, well, virtual. It costs you almost nothing until you've touched it. (Fun exercise for the reader: measure the kernel overhead for an unused 1 TiB VMA.) But creating huge spaces--that terabyte mmap wasn't theoretical--that stay untouched is hugely algorithmically useful, especially for things like malloc implementations. Why does it bother people? Two reasons. First is mlock to avoid swap. This is solvable in much better ways--I'm a fan of disabling swap in many cases anyway. Second is that, absent cgroups, it's difficult to put hard limits on memory usage in Linux. So people, looking under the streetlight, put limits on virtual usage, even though that's not what they care about limiting! Then they get angry when you break it. My refrain here, as in many cases (see for example measuring process CPU time spent in kernel mode): "X is impossible" doesn't justify Y unless Y correctly solves the problem X does. (I spent years in charge of a major memory allocator so this is a battle I've fought too many times.)

exabrial 8y ago
You cannot possibly be serious: package main func main() { for { } } Using 100mb+ of memory?
- outside1234 8y ago
  It is not "using" but allocating the addresses. The committed memory will be vastly smaller. This is how virtual memory works.
- weberc2 8y ago
  To quote the parent: `The nice thing about virtual memory is that it's, well, virtual. It costs you almost nothing until you've touched it.`
- Thaxll 8y ago
  Virtual memory is different than resident / RSS ...
- masklinn 8y ago
  > Using 100mb+ of memory? It's not using them. On modern OSX, processes get something like 2.4GB vmem by default, even if they do nothing. #include <unistd.h> int main() { for(;;) { sleep(10); } } is reported as 2377M VMEM by top/htop, on 10.11.
cyphar 8y ago
Thirdly, the kernel will kill you if your overcommit ratio is too high. I had this argument with the Go folks several years ago (when Docker would crash after starting 1000 containers because the Go runtime had allocated 8GB of virtual memory while only a tens of MB were in use and the kernel freaked out). You're right that it doesn't cost anything, other than the risk that a process can cripple your machine using its overcommitted memory mapping. And so the kernel has protections against this, which should deter language runtime developers from doing this. And let's not forget that MADV_DONTNEED is both incorrectly expensive on Linux and ridiculously expensive compared to freeing memory and reallocating it when you need it. Bryan Cantrill ranted about this for a solid half an hour in a podcast a year or two ago.
- amscanne 8y ago
  What do you mean by “free” memory? Actually unmap it? Also, I assume the crippling you’re talking about here is just the ability to rapidly apply memory pressure? Otherwise I’m very confused.
  - cyphar 8y ago
    > What do you mean by “free” memory? Actually unmap it? Sorry, I didn't phrase it well. MADV_DONTNEED is significantly more expensive than most ways that memory allocators would "free" memory. This includes just zeroing it out in userspace when necessary (so no need for a TLB modification), or simply unmapping it and remapping it when needed. > Also, I assume the crippling you’re talking about here is just the ability to rapidly apply memory pressure? Right, and if the memory is overcommitted then you can cause OOM very trivially because you already have more mapped pages than there is physical memory -- writing a byte in each page will cause intense memory pressure. Now, this doesn't mean that it would kernel panic the machine, it just means it would cause issues (OOM would figure out what process is the culprit fairly easily). This is why vm.overcommit_ratio exists (which is what I was talking about when it comes to killing a machine) -- though I just figured out that not all Linux machines ship with vm.overcommit_memory=2 (which I'm pretty sure is what SUSE and maybe some other distros ship because this is definitely an issue we've had for several years...). There's also RLIMIT_AS, which applied regardless of overcommit_memory.
toast0 8y ago
The thing is, people want a way to measure and control the amount of memory that a process uses or is likely to use. Resident memory is one way to measure actually used memory, but from the man 3 vlimit, RLIMIT_RSS is only available on linux 2.4.x, x < 30; which nobody in their right mind is still running. So we have RLIMIT_AS which limits virtual memory, or we have the default policy of hope the OOM killer kills the right thing when you run out of ram. That you have to keep fighting this battle is an indication that people's needs (or desires) aren't being well met.
cesarb 8y ago
There's a third reason: trying to allocate too much virtual memory on machines with limited physical memory will fail on Linux with the default setting vm.overcommit_memory=0. See for instance https://bugs.chromium.org/p/webm/issues/detail?id=78 https://bugs.chromium.org/p/webm/issues/detail?id=78
- sitkack 8y ago
  And those processes with large VM usage also have a problem doing an exec (fork/execvp pair) because of address space exhaustion.
dijit 8y ago
I used to think this. Then I deployed on windows. Virtual memory can’t exceed total physical memory (+pagefile) or else malloc() will fail. I am currently having an issue where memory is “allocated” but not used causing software crashes. Actual used memory is 60% of that.
- TimJYoung 8y ago
  The page file in Windows can grow and the max size, I believe, is 3 times the amount of physical memory in the machine. So, if you're trying to commit more than [Physical Memory x 4] bytes, then yes, it will fail. But, more than likely, you'll get malloc failures long before that due to address space fragmentation (unless you're doing one huge chunk).
  - Strom 8y ago
    The automatic size management doesn't go over 3x RAM, but manual configuration allows for a maximum page file size of 16TB. https://blogs.technet.microsoft.com/markrussinovich/2008/11/17/pushing-the-limits-of-windows-virtual-memory/ https://blogs.technet.microsoft.com/markrussinovich/2008/11/...
umanwizard 8y ago
Not sure whether anyone is writing iOS apps in go, but iOS refuses to allocate more than a relatively small amount of address space to each process (a few gigs, even on 64-bit devices).
martincmartin 8y ago
Great points. A third reason is core files: That 1 TB of unused virtual memory will be written out to the core file, which will take forever and/or run out of disk. This is part of the problem of running with the address sanitizer: you don't get core files on crashing, because they'd be too big.
- the8472 8y ago
  Shouldn't that be a sparse file?
networkimprov 8y ago
Re "unreasonable" I simply pasted the title of the Github issue :-)

by ahh 8y ago

exabrial 8y ago

You cannot possibly be serious: package main func main() { for { } } Using 100mb+ of memory?

outside1234 8y ago
It is not "using" but allocating the addresses. The committed memory will be vastly smaller. This is how virtual memory works.
weberc2 8y ago
To quote the parent: `The nice thing about virtual memory is that it's, well, virtual. It costs you almost nothing until you've touched it.`
Thaxll 8y ago
Virtual memory is different than resident / RSS ...
masklinn 8y ago
> Using 100mb+ of memory? It's not using them. On modern OSX, processes get something like 2.4GB vmem by default, even if they do nothing. #include <unistd.h> int main() { for(;;) { sleep(10); } } is reported as 2377M VMEM by top/htop, on 10.11.

cyphar 8y ago

Thirdly, the kernel will kill you if your overcommit ratio is too high. I had this argument with the Go folks several years ago (when Docker would crash after starting 1000 containers because the Go runtime had allocated 8GB of virtual memory while only a tens of MB were in use and the kernel freaked out). You're right that it doesn't cost anything, other than the risk that a process can cripple your machine using its overcommitted memory mapping. And so the kernel has protections against this, which should deter language runtime developers from doing this. And let's not forget that MADV_DONTNEED is both incorrectly expensive on Linux and ridiculously expensive compared to freeing memory and reallocating it when you need it. Bryan Cantrill ranted about this for a solid half an hour in a podcast a year or two ago.

amscanne 8y ago
What do you mean by “free” memory? Actually unmap it? Also, I assume the crippling you’re talking about here is just the ability to rapidly apply memory pressure? Otherwise I’m very confused.
- cyphar 8y ago
  > What do you mean by “free” memory? Actually unmap it? Sorry, I didn't phrase it well. MADV_DONTNEED is significantly more expensive than most ways that memory allocators would "free" memory. This includes just zeroing it out in userspace when necessary (so no need for a TLB modification), or simply unmapping it and remapping it when needed. > Also, I assume the crippling you’re talking about here is just the ability to rapidly apply memory pressure? Right, and if the memory is overcommitted then you can cause OOM very trivially because you already have more mapped pages than there is physical memory -- writing a byte in each page will cause intense memory pressure. Now, this doesn't mean that it would kernel panic the machine, it just means it would cause issues (OOM would figure out what process is the culprit fairly easily). This is why vm.overcommit_ratio exists (which is what I was talking about when it comes to killing a machine) -- though I just figured out that not all Linux machines ship with vm.overcommit_memory=2 (which I'm pretty sure is what SUSE and maybe some other distros ship because this is definitely an issue we've had for several years...). There's also RLIMIT_AS, which applied regardless of overcommit_memory.

toast0 8y ago

The thing is, people want a way to measure and control the amount of memory that a process uses or is likely to use. Resident memory is one way to measure actually used memory, but from the man 3 vlimit, RLIMIT_RSS is only available on linux 2.4.x, x < 30; which nobody in their right mind is still running. So we have RLIMIT_AS which limits virtual memory, or we have the default policy of hope the OOM killer kills the right thing when you run out of ram. That you have to keep fighting this battle is an indication that people's needs (or desires) aren't being well met.

cesarb 8y ago

There's a third reason: trying to allocate too much virtual memory on machines with limited physical memory will fail on Linux with the default setting vm.overcommit_memory=0. See for instance https://bugs.chromium.org/p/webm/issues/detail?id=78 https://bugs.chromium.org/p/webm/issues/detail?id=78

sitkack 8y ago
And those processes with large VM usage also have a problem doing an exec (fork/execvp pair) because of address space exhaustion.

dijit 8y ago

I used to think this. Then I deployed on windows. Virtual memory can’t exceed total physical memory (+pagefile) or else malloc() will fail. I am currently having an issue where memory is “allocated” but not used causing software crashes. Actual used memory is 60% of that.

TimJYoung 8y ago
The page file in Windows can grow and the max size, I believe, is 3 times the amount of physical memory in the machine. So, if you're trying to commit more than [Physical Memory x 4] bytes, then yes, it will fail. But, more than likely, you'll get malloc failures long before that due to address space fragmentation (unless you're doing one huge chunk).
- Strom 8y ago
  The automatic size management doesn't go over 3x RAM, but manual configuration allows for a maximum page file size of 16TB. https://blogs.technet.microsoft.com/markrussinovich/2008/11/17/pushing-the-limits-of-windows-virtual-memory/ https://blogs.technet.microsoft.com/markrussinovich/2008/11/...

umanwizard 8y ago

Not sure whether anyone is writing iOS apps in go, but iOS refuses to allocate more than a relatively small amount of address space to each process (a few gigs, even on 64-bit devices).

martincmartin 8y ago

Great points. A third reason is core files: That 1 TB of unused virtual memory will be written out to the core file, which will take forever and/or run out of disk. This is part of the problem of running with the address sanitizer: you don't get core files on crashing, because they'd be too big.

the8472 8y ago
Shouldn't that be a sparse file?

networkimprov 8y ago

Re "unreasonable" I simply pasted the title of the Github issue :-)