3 ms·

For the first 5 years that I wrote C, I never used dynamic memory. (It was a hobby.) Later I found that dynamic memory allowed me to perform more operations. By

For the first 5 years that I wrote C, I never used dynamic memory. (It was a hobby.) Later I found that dynamic memory allowed me to perform more operations. By only allocating what I needed, I could process more operations in the same amount of memory. I tripled the amount of operations due to not needing to restrict myself to an arbitrary size. The Kernel has a feature that takes advantage of this same principle: memory overcommit. Programs are always asking for way more memory than they actually need. The kernel lies to those programs, and says, "yes, yes, I'll give you all that memory... sure thing..." even if it already said that to every other program. This allows more programs to continue operating since they aren't really using all that memory. When the system does run out of memory, it can swap rarely used memory out to a swap file; when that doesn't work, the application can receive an error, or the OOM killer can just kill the process. The result is more programs doing more work, and an occasional app crash. Virtual machine hypervisors do a similar trick with memory ballooning. If the guest OS has a balloon driver, the hypervisor can overcommit memory for the VM guests, and you can provision more VMs than there's actually memory for.

polio 4y ago
It seems analogous to fractional reserve banking and the free economic performance it enables by leaning on a mostly reliable statistic about empirical peak concurrent usage of a shared resource.
newZWhoDis 4y ago
>The kernel lies to those programs Doesn’t that cause the very problem it was meant to solve? Programs ask for more memory, because they can. Users run those programs, because there’s no indication anything is wrong. And programs keep requesting more memory than need because the kernel cleans up after them! Alternatively a kernel that actually gave non-malicious (?) programs the memory they requested exclusive access to would at first run worse, but long term perform better as users rightly pester app devs for apps that use excessive memory.
- ikiris 4y ago
  welcome to the tragedy of the commons.
- rpcope1 4y ago
  You've basically described the difference between how Windows handles memory allocation and how Linux handles memory allocation, each of which has its pros and cons. Not doing overcommit has it's own serious drawbacks, especially if you make use of fork().
setr 4y ago
Never really looked into it, but if the kernel overcommits memory, and multiple applications actually utilize that memory, it would occur outside the scope of malloc, right? The kernel can’t pass an error to the app because it could occur at anytime during program execution — the only option is to OOM kill the app? Only now thinking about it, doesn’t overcomitting make it impossible for programs to be well-behaved during high system load? Everything is lying to you up until the last second, when suddenly you’re dead by OOM killer. When does malloc even return errors in practice?
jameshart 4y ago
So I have a theory as to why we were able to get away with that in the past, but can't really any more. The approach of letting applications just allocate as much memory as they wanted arose in the era of primarily 32-bit operating systems. And that meant that, at a process level, in practice 'as much memory as you want' capped out at 4GB (or 2GB if you are Win32 because... Windows reasons). And so when you're running 32 bit processes on computers with single-digit-gigs of physical RAM, and multiple-digit-gigs of disk space, and single-core processors, letting processes treat memory as 'all you can eat' and swapping out excess allocations to disk is going to work out fine. Your processor can only be running one application's code at a time, and a significant fraction of that application's entire address space can all be paged into physical memory at once, with the full amount not taking up too much disk space, so for the most part you can get away with just treating 'process address space exhausted' as being as good a signal as any of 'out of memory'. But now we're in 64-bit land, address space exhaustion is no longer a thing. If you just allow programs to keep allocating RAM until they run out of 64-bit address space, to a first approximation, 0% of your application's maximum address space will ever be able to be paged into physical RAM, and you will need several exabytes of hard drive space to hold what's paged to disk. So the problem is we now don't really have a natural limit akin to the 32-bit address ceiling to impose on applications who want to allocate memory, and it means we're instead put in the position of having to come up with policies ourselves. Stuff like setting a memory limit on a docker container. And so we wind up having to actually now think about and answer the question, 'how much memory should we let each process on this system allocate?'
- 0xbadcafebee 4y ago
  As far as I'm aware, we're still getting away with it. Your kernel and every program on your computer and every server is still doing it... Setting a memory limit on docker containers comes from automation. We built new-fangled automation that can schedule multiple copies of an application across N nodes. Then we tell people they can run more and more copies of the app, for CI/CD, for test/stage/prod, for Joe's Test App, etc. All these copies are using up memory. At some point, somebody tries to deploy to prod, which needs to start new copies of the new versions. But now they can't because the paultry few nodes have now run out of memory from all these extra copies of the apps. So now we have to impose limits or we can't deploy to prod. (you might ask: "why don't we just automate expanding the nodes to accommodate more memory?" and they did. and it doesn't work well, for reasons that give me a headache) (you might also ask: "why don't they just make an OOM killer for the automated cluster?" and they did. and it doesn't work very well either, for other headachy reasons) "Software is a gas: it always expands to fit whatever container it is stored in." - Nathan Myhrvold (https://web.archive.org/web/19990202072225/http://research.microsoft.com/acm97/nm/sld026.htm https://web.archive.org/web/19990202072225/http://research.m...)

polio 4y ago

It seems analogous to fractional reserve banking and the free economic performance it enables by leaning on a mostly reliable statistic about empirical peak concurrent usage of a shared resource.

newZWhoDis 4y ago

>The kernel lies to those programs Doesn’t that cause the very problem it was meant to solve? Programs ask for more memory, because they can. Users run those programs, because there’s no indication anything is wrong. And programs keep requesting more memory than need because the kernel cleans up after them! Alternatively a kernel that actually gave non-malicious (?) programs the memory they requested exclusive access to would at first run worse, but long term perform better as users rightly pester app devs for apps that use excessive memory.

ikiris 4y ago
welcome to the tragedy of the commons.
rpcope1 4y ago
You've basically described the difference between how Windows handles memory allocation and how Linux handles memory allocation, each of which has its pros and cons. Not doing overcommit has it's own serious drawbacks, especially if you make use of fork().

setr 4y ago

Never really looked into it, but if the kernel overcommits memory, and multiple applications actually utilize that memory, it would occur outside the scope of malloc, right? The kernel can’t pass an error to the app because it could occur at anytime during program execution — the only option is to OOM kill the app? Only now thinking about it, doesn’t overcomitting make it impossible for programs to be well-behaved during high system load? Everything is lying to you up until the last second, when suddenly you’re dead by OOM killer. When does malloc even return errors in practice?

jameshart 4y ago

So I have a theory as to why we were able to get away with that in the past, but can't really any more. The approach of letting applications just allocate as much memory as they wanted arose in the era of primarily 32-bit operating systems. And that meant that, at a process level, in practice 'as much memory as you want' capped out at 4GB (or 2GB if you are Win32 because... Windows reasons). And so when you're running 32 bit processes on computers with single-digit-gigs of physical RAM, and multiple-digit-gigs of disk space, and single-core processors, letting processes treat memory as 'all you can eat' and swapping out excess allocations to disk is going to work out fine. Your processor can only be running one application's code at a time, and a significant fraction of that application's entire address space can all be paged into physical memory at once, with the full amount not taking up too much disk space, so for the most part you can get away with just treating 'process address space exhausted' as being as good a signal as any of 'out of memory'. But now we're in 64-bit land, address space exhaustion is no longer a thing. If you just allow programs to keep allocating RAM until they run out of 64-bit address space, to a first approximation, 0% of your application's maximum address space will ever be able to be paged into physical RAM, and you will need several exabytes of hard drive space to hold what's paged to disk. So the problem is we now don't really have a natural limit akin to the 32-bit address ceiling to impose on applications who want to allocate memory, and it means we're instead put in the position of having to come up with policies ourselves. Stuff like setting a memory limit on a docker container. And so we wind up having to actually now think about and answer the question, 'how much memory should we let each process on this system allocate?'

0xbadcafebee 4y ago
As far as I'm aware, we're still getting away with it. Your kernel and every program on your computer and every server is still doing it... Setting a memory limit on docker containers comes from automation. We built new-fangled automation that can schedule multiple copies of an application across N nodes. Then we tell people they can run more and more copies of the app, for CI/CD, for test/stage/prod, for Joe's Test App, etc. All these copies are using up memory. At some point, somebody tries to deploy to prod, which needs to start new copies of the new versions. But now they can't because the paultry few nodes have now run out of memory from all these extra copies of the apps. So now we have to impose limits or we can't deploy to prod. (you might ask: "why don't we just automate expanding the nodes to accommodate more memory?" and they did. and it doesn't work well, for reasons that give me a headache) (you might also ask: "why don't they just make an OOM killer for the automated cluster?" and they did. and it doesn't work very well either, for other headachy reasons) "Software is a gas: it always expands to fit whatever container it is stored in." - Nathan Myhrvold (https://web.archive.org/web/19990202072225/http://research.microsoft.com/acm97/nm/sld026.htm https://web.archive.org/web/19990202072225/http://research.m...)