4 ms·

Be aware `jemalloc` will make you suffer the observability issues of `MADV_FREE`. `htop` will no longer show the truth about how much memory is in use. * https

• a1o 3y ago
  Thank you! That was very thorough! I will be reading the links. :)
• singron 3y ago
  Note that glibc has a similar problem in multithreaded contexts. It strands unused memory in thread-local pools, which grows your memory usage over time like a memory leak. We got lower memory usage that didn't grow over time by switching to jemalloc. Example of this: https://github.com/prestodb/presto/issues/8993 https://github.com/prestodb/presto/issues/8993
• masklinn 3y ago
  The musl remark is funny, because jemalloc's use of pretty fine-grained arenas sometimes leads to better memory utilisation through reduced fragmentation. For instance Aerospike couldn't fit in available memory under (admittedly old) glibc, and jemalloc fixed the issue: http://highscalability.com/blog/2015/3/17/in-memory-computing-at-aerospike-scale-when-to-choose-and-ho.html http://highscalability.com/blog/2015/3/17/in-memory-computin... And this is not a one-off: https://hackernoon.com/reducing-rails-memory-use-on-amazon-linux-with-jemalloc https://hackernoon.com/reducing-rails-memory-use-on-amazon-l... https://engineering.linkedin.com/blog/2021/taming-memory-fragmentation-in-venice-with-jemalloc https://engineering.linkedin.com/blog/2021/taming-memory-fra... jemalloc also has extensive observability / debugging capabilities, which can provide a useful global view of the system, it's been used to debug memleaks in JNI-bridge code: https://www.evanjones.ca/java-native-leak-bug.html https://www.evanjones.ca/java-native-leak-bug.html https://technology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/ https://technology.blog.gov.uk/2015/12/11/using-jemalloc-to-...
  • nh2 3y ago
    Yes, almost everybody who looks at memory usage in production will eventually discover glibc's memory fragmentation issues. This is how I learned about this topic. Setting the env var MALLOC_MMAP_THRESHOLD_=65536 usually solves these problems instantaneously. Most programmers seem to not bother to understand what is going on (thus arriving at the above solution) but follow "we switched to jemalloc and it fixes the issue". (I have no opinion yet on whether jemalloc is better or worse than glibc malloc. Both have tunables, and will create problematic corner cases if the tunables are not set accordingly. The fact that jemalloc has /more/ tunables, and more observability / debugging features, seems like a pro point for those that read the documentation. For users that "just want low memory usage", both libraries' defaults look bad, and the musl attitude seems like the best default, since OOM will cause a crash vs just having the program be some percent slower.)
• dralley 3y ago
  glibc isn't totally free of such issues https://www.algolia.com/blog/engineering/when-allocators-are-hoarding-your-precious-memory/ https://www.algolia.com/blog/engineering/when-allocators-are...
• the8472 3y ago
  Aiming to please people who panic about their RSS numbers seems... misguided? It seems like worrying about RAM being "used" as file cache[0]. If you want to gauge whether your system is memory-limited look at the PSI metrics instead. [0] https://www.linuxatemyram.com/ https://www.linuxatemyram.com/
  • nh2 3y ago
    Those are not the same. You can see cache usage in htop; it has a different colour. With MADV_FREE, it looks like the process is still using the memory. That sucks: If you have some server that's slow, you want to SSH into a server and see how much memory each process takes. That's a basic, and good, observability workflow. Memory leaks exist, and tools should show them easily. The point of RES is to show resident memory, not something else. If you change htop to show the correct memory, that'd fix the issue of course.
    • the8472 3y ago
      Well, RES is resident in physical memory. It just is marked so that the kernel can make reclaim it when it needs to. But until then it's resident. If you want to track leaks you need resident-and-in-use metric which may be more difficult to come by (probably requires scanning smaps?). It's a case of people using the subtly wrong metrics and then trying to optimize tools chasing that metric rather than improving their metrics. That's what I'm calling misguided.
• saagarjha 3y ago
  Not that I would recommend using jemalloc by default but it’s definitely going to be better than musl’s allocator ;)