3 ms·
I liked the article. It was a simplified explanation, which is a good thing in its own right: not everyone reading it cares about the gory details. I'll prov
by amalcon 15y ago
I liked the article. It was a simplified explanation, which is a good thing in its own right: not everyone reading it cares about the gory details. I'll provide some such details for the curious, though.
Technically, malloc() is not a system call; it is a library function that calls either brk()/sbrk() or mmap() (or just uses memory it already "has"). The choice depends on the situation and just what it's doing.
mmap() engages the VM subsystem, backing a chunk of RAM with part of a given file on disk (in this case with a special flag to not actually do that, but just give the RAM). Of course this will never require actual RAM until you go to use it: that's the whole point of the call. Normally mmap() will not cause OOM issues, because it creates exactly as much virtual memory out of thin air as it requests physical memory. That's not true in this usage, though. I suppose in this usage, it could block off a chunk of virtual memory.
brk() and sbrk() are the more "normal" ways to get extra memory, by expanding the heap. The new memory must be initialized manually (or not, as the runtime prefers). The main reason to not do this all the time is that the heap itself must be contiguous. Using mmap() lets malloc() return pages to the system under more varied conditions and work around parts of the address space that contain other things[1].
Calls to mmap()/brk()/sbrk() are rather expensive compared to malloc() calls. They are system calls, after all. That's why programs will normally ask for more memory than they actually need at the moment: it reduces the amortized runtime cost of malloc(). Linux tries to take advantage of this.
So, the strange Linux behavior is that if a process calls mmap()/brk()/sbrk() (and doesn't hit the addressable limit) Linux will simply record that the process has requested mapping of a particular region of memory. Then, later, the VM subsystem hands out the memory when the process tries to do something to it. The question is, what to do if there isn't any memory to hand out?
There's no way to cause an arbitrary memory access to fail with ENOMEM, so Linux has this hacky workaround where it semi-arbitrarily selects and kills something.
The historical way to avoid this was to have a huge amount of virtual memory, hence the suggestions like "twice the amount of RAM in the system". The system would become unusably slow before the OOM killer would engage, and the user would manually kill some relatively unimportant processes.
[1]- In older versions of the Linux kernel, the program code would be stored in memory about 1G away from the heap. So, as soon as you used about 1G, you'd run into your code and malloc() would no longer be able to brk() for more memory.
- acqq 15y agoThe post linked from the article is really nice: http://lwn.net/Articles/104185/ http://lwn.net/Articles/104185/ "in rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.)" I always thought that mapping of virtual memory should succeed only if there is enough virtual memory available (i.e. when not in RAM than in page file). Why is the overcommit concept better? Because malloc takes way too much VM unnecessary? Then why not just correcting malloc?
- amalcon 15y agoWell, malloc() is very much doing the right thing in a world where (virtual) memory is plentiful. The performance of the program improves substantially, without impacting memory usage that much. I never understood why the system doesn't do the accounting strictly, even if it waits to actually hand out individual pages. Then, it could give an error on brk() or mmap() when the memory would be "overbooked". Memory "saved" this way would be unusable to other processes, but it would be usable as buffer cache, and fork() would still be fast. edit: Another possible solution would be a special "out-of-memory" signal. The program could trap that signal and do the right thing with it. The problem with that is that it's difficult to program for out-of-memory conditions. Something as simple as printf() won't work at all, and could cause a signal loop depending on implementation. Given that, almost no programs would do it, so there would be little benefit.
- gsg 15y agoWhy would you deny allocation requests larger than immediately available physical memory when they could be serviced perfectly well by paging? On modern systems malloc and friends manage address space, not physical memory. Failure should only be reported if that resource is exhausted.