4 ms·

Author here. I measured RSS with swap disabled, so physical memory usage. > Minor quibble: it's not accurate to call `malloc()` and family in glibc the "operat

Author here. I measured RSS with swap disabled, so physical memory usage. > Minor quibble: it's not accurate to call `malloc()` and family in glibc the "operating system's memory allocator." I know. I just explained it like that in order to keep the material digestible to a wider audience.

usr1106 8y ago
Thank you for the clarification. > I measured RSS with swap disabled, so physical memory usage. That sounds correct to me. More details you could get from /proc/<pid>/smaps. There are a couple of other issues I found confusing in your text. If you intentionally make simplifications, I would recommend to add at least a footnote to indicate so. Linux has only 1 heap. But while the heap is declared to be of a certain maximum size using brk() it does not mean that all of the pages are really in RAM (as before, swapping aside). A page gets only really allocated when it is accessed the first time. And it can get deallocated again using madvise() if the program know that it is no longer needed. malloc_trim() calls madvise(). So you are correct, it does not only move the top of the heap. Probably it was that way many years ago. So in general the Linux heap can be sparsely mapped to RAM. But if you use RSS your measuring takes that into account already. The arenas are a feature of glibc. They are documented (to some degree) in the man pages, so I would not call them magic. Also malloc_info() prints how they are used. It appears to me that arena 0 is on the Linux heap. If a program is multithreaded, glibc will call mmap() to create an additional memory area to be used for the additional arenas. While avoiding mutex contention might be a good thing if the threads are allocating many small blocks, the overhead for arenas might not be justified for rarely allocating bigger areas as the Ruby heap implementation probably does it. So limiting the number of arenas might indeed be beneficial, it's just a typical trade-off one needs to understand. Not that that is easy, but magic is the wrong description IMHO. Let's call the glibc functionality above the system allocator (I'm not sure whether this is the correct name or whether it even has an official name). However, glibc has yet another completely different allocator, the mmap allocator. For allocations bigger than MMAP_THRESHOLD glibc will not use the heap or any of the additional arenas, but just directly make a completely new memory mapping from the Linux kernel. Again these will lazily/sparsely allocated to RAM. The mmap allocator does not use arenas at all. I am not a regular Ruby user, so I don't know whether the Ruby heap uses glibcs's system allocator or the mmap allocator or what mix of both. (Mixing them is transparent to a program.) However, from your description I would guess that it mostly uses the system allocator, otherwise arenas would not make any change and AFAIK malloc_trim() has no effect on the mmap allocator at all. Have you considered either changing the setting of MMAP_THRESHOLD or the Ruby interpreter code such, that the Ruby heap uses only the mmap allocator? It appears to me that having the Ruby heap management on top of the glibc system allocator is just not a good idea. The system allocator tries to be a good compromise for a widely varying spectrum of applications. But the Ruby heap management is one very specific case, partially duplicating the work that the glibc allocator does. I'd guess having the Ruby heap running closer to the operating system should improve things. The mmap allocator of glibc might be an easy way to achieve that. Otherwise the Ruby head should use mmap() directly, because it should know best how it wants to use the memory. But that would be much more difficult to implement of course. P.S. Your visualizer looks impressive. Unfortunately I did not have time to study it in detail. Did you make sure that the code does never access a page that has been mapped but never accessed by Ruby? Because that would dirty the page and increase the RSS.
- FooBarWidget 8y ago
  > Did you make sure that the code does never access a page that has been mapped but never accessed by Ruby? Because that would dirty the page and increase the RSS. Dirtying only happens upon write. My visualizer never writes, only reads.

usr1106 8y ago

Thank you for the clarification. > I measured RSS with swap disabled, so physical memory usage. That sounds correct to me. More details you could get from /proc/<pid>/smaps. There are a couple of other issues I found confusing in your text. If you intentionally make simplifications, I would recommend to add at least a footnote to indicate so. Linux has only 1 heap. But while the heap is declared to be of a certain maximum size using brk() it does not mean that all of the pages are really in RAM (as before, swapping aside). A page gets only really allocated when it is accessed the first time. And it can get deallocated again using madvise() if the program know that it is no longer needed. malloc_trim() calls madvise(). So you are correct, it does not only move the top of the heap. Probably it was that way many years ago. So in general the Linux heap can be sparsely mapped to RAM. But if you use RSS your measuring takes that into account already. The arenas are a feature of glibc. They are documented (to some degree) in the man pages, so I would not call them magic. Also malloc_info() prints how they are used. It appears to me that arena 0 is on the Linux heap. If a program is multithreaded, glibc will call mmap() to create an additional memory area to be used for the additional arenas. While avoiding mutex contention might be a good thing if the threads are allocating many small blocks, the overhead for arenas might not be justified for rarely allocating bigger areas as the Ruby heap implementation probably does it. So limiting the number of arenas might indeed be beneficial, it's just a typical trade-off one needs to understand. Not that that is easy, but magic is the wrong description IMHO. Let's call the glibc functionality above the system allocator (I'm not sure whether this is the correct name or whether it even has an official name). However, glibc has yet another completely different allocator, the mmap allocator. For allocations bigger than MMAP_THRESHOLD glibc will not use the heap or any of the additional arenas, but just directly make a completely new memory mapping from the Linux kernel. Again these will lazily/sparsely allocated to RAM. The mmap allocator does not use arenas at all. I am not a regular Ruby user, so I don't know whether the Ruby heap uses glibcs's system allocator or the mmap allocator or what mix of both. (Mixing them is transparent to a program.) However, from your description I would guess that it mostly uses the system allocator, otherwise arenas would not make any change and AFAIK malloc_trim() has no effect on the mmap allocator at all. Have you considered either changing the setting of MMAP_THRESHOLD or the Ruby interpreter code such, that the Ruby heap uses only the mmap allocator? It appears to me that having the Ruby heap management on top of the glibc system allocator is just not a good idea. The system allocator tries to be a good compromise for a widely varying spectrum of applications. But the Ruby heap management is one very specific case, partially duplicating the work that the glibc allocator does. I'd guess having the Ruby heap running closer to the operating system should improve things. The mmap allocator of glibc might be an easy way to achieve that. Otherwise the Ruby head should use mmap() directly, because it should know best how it wants to use the memory. But that would be much more difficult to implement of course. P.S. Your visualizer looks impressive. Unfortunately I did not have time to study it in detail. Did you make sure that the code does never access a page that has been mapped but never accessed by Ruby? Because that would dirty the page and increase the RSS.

FooBarWidget 8y ago
> Did you make sure that the code does never access a page that has been mapped but never accessed by Ruby? Because that would dirty the page and increase the RSS. Dirtying only happens upon write. My visualizer never writes, only reads.