3 ms·
This assumes glibc can actually shuffle memory locations around, which it can’t. Except for some special cases you can’t return memory to the kernel, you can on
by tinus_hn 7y ago
This assumes glibc can actually shuffle memory locations around, which it can’t. Except for some special cases you can’t return memory to the kernel, you can only stop using it and hope it’s swapped out. Perhaps you could zero it so it compresses well.
- chrisseaton 7y agoYou can unmap memory.
- saagarjha 7y agoI mean you can munmap a page if it's not being used…
- knome 7y agomadvise( start, size, MADV_DONTNEED ); start must be page-aligned. size is in bytes, but if you take one byte of a page, you take the whole thing, so clip as appropriate. this informs the kernel you're done with some part of the memory. mapped things will be re-read and others will act as if zero'd when later accessed. both can be safely dropped without taking space in the swap, instead being newly created when next needed.
- kevinoid 7y agoAlthough this is true on Linux, it's worth noting that this behavior doesn't match the specified behavior of posix_madvise.[1] It was also a subject of Bryan Cantril's hilarious "A crime against common sense" Surge 2015 talk (at ~1:06:50).[2] [1]: https://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_madvise.html https://pubs.opengroup.org/onlinepubs/9699919799/functions/p... [2]: https://youtu.be/bg6-LVCHmGM?t=3518 https://youtu.be/bg6-LVCHmGM?t=3518
- jacobush 7y agoAh, so that's where we are now. It used to be that we didn't clear the memory for performance reasons. Now, we clear the memory for performance reasons. Very entropy. I think I like it.
- cozzyd 7y agoit clearly can if malloc_trim works? I guess I didn't see how long that operation takes though, but it could certainly be useful for long-running processes like web browsers to call it every once in a while...
- CJefferson 7y agoNo, it definately can't. In C one would usually store pointers into memory blocks, which it is impossible to "fix up". What you can do is search through all allocated memory for full 4K pages which are unused, which can be "given back" to the OS. However, it's unclear how much benefit there is from doing this. If you give back memory too eagerly, and then need to request more later, you end up thrashing the kernel. Assuming you have some swap space, this unused memory gets swapped out. It could be (hard to tell, but I do this in my own memory managers) that the memory is technically still assigned to the program, but the pages are marked as unused, in which point there is no "memory" there at all, just address space which you can start using again later. However, it can still look like my program is using something crazy like 1TB of memory.
- chrisseaton 7y ago> No, it definately can't. Glibc malloc can return memory to the OS. This is what the trim that people are talking about does.
- scott_s 7y agoI got tripped up on that statement, too. The problem is that tinus_hn's parent comment started with a false premise: that the only way to return memory to the OS is by moving memory around, which is not the case. I think CJefferson was saying that manual memory allocators can't move memory around (which in the general case is true, but you point out in a different thread there are cases when it can), because the rest of their comment is explaining how they can return memory.
- CJefferson 7y agoSorry, I should have been clear I was specifically replying to suggestion that glibc (or any malloc, by which I mean C standard library memory manager) could move memory allocations.
- rmind 7y agoUmm.. that's not really true. Most userspace memory allocators are based on mmap(). Using sbrk() is legacy. The slab allocator (Bonwick, 1994) is the dominant algorithm (both in userspace and the UNIX-like kernels), although there are various flavours and somewhat different implementations of it. Since it uses fixed-sized allocations under the hood, they are nicely packed in pages. Those pages can be released with munmap() once there are no used blocks. Sure, depending on the application and workload, allocations can be become quite distributed amongst the pages over the time, but that is a separate problem. Also, as another comment states: madvise() is also an option, but it doesn't reclaim the virtual address space. On 32-bit systems VA space exhaustion can be a problem, especially with "modern" applications.
- rsecora 7y agoAnswers like this is the reason I came back to HN. Facts and to the point.
- wahern 7y agoBoth glibc and musl libc still uses brk for most allocations. You can see it when strace'ing simple programs. After the linker--which uses mmap for some allocations as it can't use malloc--has finished you'll see brk calls to satisfy application allocations. Also you can verify in the code, e.g. https://git.musl-libc.org/cgit/musl/tree/src/malloc/expand_heap.c?id=55a1c9c8#n56 https://git.musl-libc.org/cgit/musl/tree/src/malloc/expand_h... I believe OpenBSD's malloc only uses mmap and also aggressively munmap's memory--to better catch application bugs and to keep addresses randomized.