8 ms·

When allocators are hoarding your precious memory (2021)

94 points by marcodiego 3y ago

vlovich123 3y ago
TLDR glibc’s malloc doesn’t work well for server workloads. Nice deep dive into the available options to trim it, but the FUD around switching to something like mimalloc or the most recent tcmalloc is unwarranted considering both have seen large scale deployments and have same defaults out of the gate that don’t need tuning (and any tuning you do for glibc would need similar validation requirements).
- hinkley 3y ago
  In early days Java’s allocator was quite a bit faster than C++ in part because memory allocation was not fully concurrent, making allocation part of the sequential element of Amdahl’s law. Where we are now is better, but has its own problems.
  - masklinn 3y ago
    > Where we are now is better Not really. Lots of rust users come from high level languages so when people come in and complain about rust being unexpectedly slow allocation issues are in the top 5 at least. It’s incredible how shit the standard allocators are in almost all systems.
    - athanagor2 3y ago
      Isn’t this inevitable if the user allocates each time a struct/array is created, rather than allocating upfront?
      - masklinn 3y ago
        Meh. Users not used to tracking allocations keep to the same regime (lots of allocations), but because the platform allocators are utterly terrible the allocation overhead is orders of magnitude higher than in even a relatively basic runtime, thus the program is dog slow. This is not inevitable, platforms could provide allocators which are less awful. Obviously they can’t be as fast as specialised runtime facilities but when you see the gains many applications get by just swapping in jemalloc or some such…
        PH95VuimJjqBqy 3y ago
        I would simplify it and say it's an issue with developers who don't understand manual memory management (allocating memory pools up front is just 1 such strategy for manual memory management).
        masklinn 3y ago
        That’s not a simplification, that’s a misattribution. It’s not the beginner’s fault that most system allocators suck so hard you must do your utmost to limit their use.
        hinkley 3y ago
        Otherwise known as Victim Blaming. C developers are that old guy who white knuckles everything and calls people weak for wanting help. Many of the features in modern programs didn’t exist in the 90’s not because they hadn’t been thought of but because people spent all their energy getting the first fifty features to work reliably. Every app had a couple cool features. Most of them are de rigeur now because we can.
        PH95VuimJjqBqy 3y ago
        This is akin to the idea that a compiler will eventually solve performance problems. We're still waiting on that to happen. There's a difference between a developer who understands manual memory management choosing to use a tech that does it for you and a developer choosing to use a tech that does it for you so they don't have to learn manual memory management. manual memory management is table stakes for any halfway competent developer.
        wongarsu 3y ago
        Rust used to use jemalloc, which was arguably the better default. For some reason they switched it to the system allocator when they added support for specifying the global allocator
        steveklabnik 3y ago
    - saagarjha 3y ago
      They’re not. They are supposed to be general-purpose allocators with requirements that may not match your workload.
      - masklinn 3y ago
        > They’re not. Are too > They are supposed to be general-purpose allocators with requirements that may not match your workload. I don’t think they match any workload which exists anywhere anymore. Possibly their workload existed back in the 90s. Jemalloc is also a general-purpose allocator, it‘s the one freebsd uses.
- jeffbee 3y ago
  glibc's malloc doesn't work well for any workload. If your workload is anything other than trivial you should be using another one. > FUD around switching to something like mimalloc or the most recent tcmalloc Certainly, and the specific FUD about libc++ is unwarranted for tcmalloc, considering the developer of tcmalloc also uses libc++.
  - spockz 3y ago
    So which Malloc should we be using? How about setting that as a default? If apparently the trade off is so cookie cutter clear.
    - jeffbee 3y ago
      If you have a non-trivial workload (i.e. one that costs money) you should shop around and use the one that is the most efficient in your application.
    - kelnos 3y ago
      I assume what the GP actually meant was that glibc's allocator isn't tuned for any specific workloads. It tries to be a jack-of-all-trades, and in doing so, just ends up being okay-ish. As for what allocator you should be using, it depends on your workload. You'd need to test out some different options to see what works best. And "best" might also be something you have to define. Maybe you're ok with higher baseline memory usage if allocations are faster. Or maybe you want the lower memory usage.
      - vlovich123 3y ago
        Actually, mimalloc and tcmalloc both show that across almost any workload they outperform glibc. It’s difficult to outperform everywhere of course, but in terms of “works best without knowing the workload a priori”, tcmalloc and mimalloc should handily trounce glibc. Know if you know your workload and start tuning glibc, maybe it can outperform? Not sure but I’d also be skeptical. Tcmalloc and mimalloc both apply really advanced techniques that also require newer kernel support to implement user side RCU iirc. Glibc’s allocator by comparison is much longer in the tooth and just can’t compete on that. Most workloads don’t care of course, but glibc’s penchant for hanging onto RAM when not needed is one very user visible consequence of this.
pwagland 3y ago
We once ran into a very similar issue, but using the old glibc allocator in bionic. For us the difference was between JVM versions/vendors. The arenas in glibc all allocated memory, based on what thread they are in. Because threads in Java are somewhat ubiquitous, then this caused massive "memory leaks", read allocated memory that wasn't being used, as every allocator was getting used, which caused them to all build up their buffers. This in turn caused a large increase in AWS bills, as we had to increase the memory allocated to avoid the OOMK. We ended up moving to tcmalloc, mostly because the Debian packages worked the best for us. we did testing between the different malloc replacements, and honestly, there wasn't a huge difference between them for our workloads, although they _did_ have differences, with minor variations in peak/avg memory usage and CPU usage.
perlgeek 3y ago
This vaguely reminded of the time where our self-hosted installation of JVM-based app worked fine in DEV and QA (on vmware-based VMs), but in PROD it would error out the time with "out of memory", even though the machines (bare metal) had far more RAM. It turned out they also had lots more CPU cores (they were repurposed former DB machines), and by default the memory manager (of the JVM, I suppose?) scaled the allocation block size the number of CPUs, so it tried to allocate far more memory than necessary, and since this was still a 32-bit JVM it quickly ran out of space for its allocations. Once we found out what the issue was, it was pretty easy to tune with an environment variable.
- kevingadd 3y ago
  32 bit and high core count is definitely a nasty combo. Many applications spawn 2-4 threads per core which is a great way to eat up that address space.
  - nine_k 3y ago
    But don't all threads of a process share mostly the same address space? I could imagine that starting many processes would exhaust address space, or allocating a lot of virtual memory / mmap without committing.
    - simcop2387 3y ago
      That's one of the fun bits, other processes don't share the memory space. That's one reason you can have 16GiB of ram on a 32bit x86 server without much issue. This was done on x86 through a tech called PAE (Physical Address Extension) that let the OS know there was more memory and allocate it around, but any one process was limited to 4GiB (usually lower due to kernel/userspace barriers) because of 32 bit pointers. So if those threads all allocated memory it was easier to hit the limit but if you used separate processes then they could each reach 4GiB (or whatever the limit is) independently without causing problems.
- jon-wood 3y ago
  Off topic, so apologies in advance. Why do environments so often get capitalised like that? Prod isn’t an acronym, it’s an abbreviation of Production. Similarly Dev is an abbreviation of Development. I know in the grand scheme of things it’s meaningless, this is just one of those things that’s like someone scratching their fingers down a blackboard when I see it day to day.
theolivenbaum 3y ago
Interesting, we had very similar issues with malloc and RocksDB on a production instance, ended up switching to jemalloc and the problem went away. Cloudflare had some similar issues with it as well: https://blog.cloudflare.com/the-effect-of-switching-to-tcmalloc-on-rocksdb-memory-use/ https://blog.cloudflare.com/the-effect-of-switching-to-tcmal...
dilyevsky 3y ago
They better hurry up with that - i think 16.04 only getting secuirty patches until the end of this year without enterprise license
- SoftTalker 3y ago
  It's already past EOL unless you're paying for an "Ubuntu Pro" subscription. As is 18.04.
ceeam 3y ago
Well, Linus told you ages ago that the "fundamental design principle of Linux is to have fun". So they change stuff like this for fun. Isn't it fun for you too? And I will stick to FreeBSD, thank you. And yes, I understand that technically glibc is not a part of "Linux", but...
com2kid 3y ago
The number of arguments I've gotten into with other engineers that "manual memory allocation is deterministic, GCs are not!" is too high. First off, GCs are quite deterministic, unless they are calling rand() someplace. I've walked through GC code under controlled conditions with identical repeated allocations, GCs do the exact same thing under the exact same circumstances every time. Second off, you have to pay the price of allocation and deal with memory fragmentation somewhere. GCs typically pay the price at the tail end (deallocation, compaction) but have absurdly fast allocators (a handful of instructions), where as manual memory allocators pay the price for allocation (finding a slab of memory) up front, and unless you know how to work around the details of your allocator, you end up dealing with fragmentation yourself. FWIW this also means many, many, types of applications can gain a boost from intelligent memory allocation strategies. There is the famous example of the cruise missile that never worried about deallocation because it would explode before it ran out of memory, but scenarios involving bespoke memory management being better than "just use libc!" are actually quite common. Have a stateless microservice? If you know the max size of memory it'll use, for each connection that comes in just allocate a slab of memory and when the connection closes free the entire slab, far more efficient than using a GC or any standard allocator. Also having to calculate the max amount of memory a single connection can use is a great way to figure out how much load an instance can handle (depending if you are CPU bound or not). For certain stages of compilation, compilers can get away with not freeing memory and just exiting and letting the OS reclaim everything. (Of course less doable now days with compilers running as background services recompiling code as it is typed, and also whole program optimization has made it so compilers use a lot more memory, so I don't know if this strategy is still in use in these modern times!) The tl;dr is that freeing memory is complicated no matter if you have a GC or not. It is something every developer should be thinking about from time to time.
- Karellen 3y ago
  I thought that GCs are often run in their own thread, and because scheduling at the OS layer is seen as non-deterministic from the application's viewpoint (you never know when your own threads are going to get stopped/started), you can't control exactly when the GC is run? Is that wrong? Or obsolete? Or GC-dependent? But also, just because something is theoretically deterministic (e.g. it might technically be possible to work out when a GC sweep will run, and multiple runs of the exact same program will cause the exact same set of GC sweeps), actually working out when a GC sweep will run is so non-trivial the only way to actually work it out is to run the program and watch it happen, and also it becomes impossible to predict how any changes at all to a program will affect how the GC runs as a result (other than making the change and running it). Or are there ways to deduce when GC sweeps will run for a given program, without actually running it?
SilasHaslam 3y ago
I read the headline as: When alligators are boarding your precious memories. Sigh..
kelnos 3y ago
It was frustrating that the article ended with no real resolution. It was posted almost a year and a half ago; I wonder if there actually is some follow-up somewhere.

94 points by marcodiego 3y ago

TLDR glibc’s malloc doesn’t work well for server workloads. Nice deep dive into the available options to trim it, but the FUD around switching to something like mimalloc or the most recent tcmalloc is unwarranted considering both have seen large scale deployments and have same defaults out of the gate that don’t need tuning (and any tuning you do for glibc would need similar validation requirements).

hinkley 3y ago
In early days Java’s allocator was quite a bit faster than C++ in part because memory allocation was not fully concurrent, making allocation part of the sequential element of Amdahl’s law. Where we are now is better, but has its own problems.
- masklinn 3y ago
  > Where we are now is better Not really. Lots of rust users come from high level languages so when people come in and complain about rust being unexpectedly slow allocation issues are in the top 5 at least. It’s incredible how shit the standard allocators are in almost all systems.
  - athanagor2 3y ago
    Isn’t this inevitable if the user allocates each time a struct/array is created, rather than allocating upfront?
    - masklinn 3y ago
      Meh. Users not used to tracking allocations keep to the same regime (lots of allocations), but because the platform allocators are utterly terrible the allocation overhead is orders of magnitude higher than in even a relatively basic runtime, thus the program is dog slow. This is not inevitable, platforms could provide allocators which are less awful. Obviously they can’t be as fast as specialised runtime facilities but when you see the gains many applications get by just swapping in jemalloc or some such…
      - PH95VuimJjqBqy 3y ago
        I would simplify it and say it's an issue with developers who don't understand manual memory management (allocating memory pools up front is just 1 such strategy for manual memory management).
        masklinn 3y ago
        That’s not a simplification, that’s a misattribution. It’s not the beginner’s fault that most system allocators suck so hard you must do your utmost to limit their use.
        hinkley 3y ago
        Otherwise known as Victim Blaming. C developers are that old guy who white knuckles everything and calls people weak for wanting help. Many of the features in modern programs didn’t exist in the 90’s not because they hadn’t been thought of but because people spent all their energy getting the first fifty features to work reliably. Every app had a couple cool features. Most of them are de rigeur now because we can.
        PH95VuimJjqBqy 3y ago
        This is akin to the idea that a compiler will eventually solve performance problems. We're still waiting on that to happen. There's a difference between a developer who understands manual memory management choosing to use a tech that does it for you and a developer choosing to use a tech that does it for you so they don't have to learn manual memory management. manual memory management is table stakes for any halfway competent developer.
      - wongarsu 3y ago
        Rust used to use jemalloc, which was arguably the better default. For some reason they switched it to the system allocator when they added support for specifying the global allocator
        steveklabnik 3y ago
  - saagarjha 3y ago
    They’re not. They are supposed to be general-purpose allocators with requirements that may not match your workload.
    - masklinn 3y ago
      > They’re not. Are too > They are supposed to be general-purpose allocators with requirements that may not match your workload. I don’t think they match any workload which exists anywhere anymore. Possibly their workload existed back in the 90s. Jemalloc is also a general-purpose allocator, it‘s the one freebsd uses.
jeffbee 3y ago
glibc's malloc doesn't work well for any workload. If your workload is anything other than trivial you should be using another one. > FUD around switching to something like mimalloc or the most recent tcmalloc Certainly, and the specific FUD about libc++ is unwarranted for tcmalloc, considering the developer of tcmalloc also uses libc++.
- spockz 3y ago
  So which Malloc should we be using? How about setting that as a default? If apparently the trade off is so cookie cutter clear.
  - jeffbee 3y ago
    If you have a non-trivial workload (i.e. one that costs money) you should shop around and use the one that is the most efficient in your application.
  - kelnos 3y ago
    I assume what the GP actually meant was that glibc's allocator isn't tuned for any specific workloads. It tries to be a jack-of-all-trades, and in doing so, just ends up being okay-ish. As for what allocator you should be using, it depends on your workload. You'd need to test out some different options to see what works best. And "best" might also be something you have to define. Maybe you're ok with higher baseline memory usage if allocations are faster. Or maybe you want the lower memory usage.
    - vlovich123 3y ago
      Actually, mimalloc and tcmalloc both show that across almost any workload they outperform glibc. It’s difficult to outperform everywhere of course, but in terms of “works best without knowing the workload a priori”, tcmalloc and mimalloc should handily trounce glibc. Know if you know your workload and start tuning glibc, maybe it can outperform? Not sure but I’d also be skeptical. Tcmalloc and mimalloc both apply really advanced techniques that also require newer kernel support to implement user side RCU iirc. Glibc’s allocator by comparison is much longer in the tooth and just can’t compete on that. Most workloads don’t care of course, but glibc’s penchant for hanging onto RAM when not needed is one very user visible consequence of this.

pwagland 3y ago

We once ran into a very similar issue, but using the old glibc allocator in bionic. For us the difference was between JVM versions/vendors. The arenas in glibc all allocated memory, based on what thread they are in. Because threads in Java are somewhat ubiquitous, then this caused massive "memory leaks", read allocated memory that wasn't being used, as every allocator was getting used, which caused them to all build up their buffers. This in turn caused a large increase in AWS bills, as we had to increase the memory allocated to avoid the OOMK. We ended up moving to tcmalloc, mostly because the Debian packages worked the best for us. we did testing between the different malloc replacements, and honestly, there wasn't a huge difference between them for our workloads, although they _did_ have differences, with minor variations in peak/avg memory usage and CPU usage.

perlgeek 3y ago

This vaguely reminded of the time where our self-hosted installation of JVM-based app worked fine in DEV and QA (on vmware-based VMs), but in PROD it would error out the time with "out of memory", even though the machines (bare metal) had far more RAM. It turned out they also had lots more CPU cores (they were repurposed former DB machines), and by default the memory manager (of the JVM, I suppose?) scaled the allocation block size the number of CPUs, so it tried to allocate far more memory than necessary, and since this was still a 32-bit JVM it quickly ran out of space for its allocations. Once we found out what the issue was, it was pretty easy to tune with an environment variable.

kevingadd 3y ago
32 bit and high core count is definitely a nasty combo. Many applications spawn 2-4 threads per core which is a great way to eat up that address space.
- nine_k 3y ago
  But don't all threads of a process share mostly the same address space? I could imagine that starting many processes would exhaust address space, or allocating a lot of virtual memory / mmap without committing.
  - simcop2387 3y ago
    That's one of the fun bits, other processes don't share the memory space. That's one reason you can have 16GiB of ram on a 32bit x86 server without much issue. This was done on x86 through a tech called PAE (Physical Address Extension) that let the OS know there was more memory and allocate it around, but any one process was limited to 4GiB (usually lower due to kernel/userspace barriers) because of 32 bit pointers. So if those threads all allocated memory it was easier to hit the limit but if you used separate processes then they could each reach 4GiB (or whatever the limit is) independently without causing problems.
jon-wood 3y ago
Off topic, so apologies in advance. Why do environments so often get capitalised like that? Prod isn’t an acronym, it’s an abbreviation of Production. Similarly Dev is an abbreviation of Development. I know in the grand scheme of things it’s meaningless, this is just one of those things that’s like someone scratching their fingers down a blackboard when I see it day to day.

theolivenbaum 3y ago

Interesting, we had very similar issues with malloc and RocksDB on a production instance, ended up switching to jemalloc and the problem went away. Cloudflare had some similar issues with it as well: https://blog.cloudflare.com/the-effect-of-switching-to-tcmalloc-on-rocksdb-memory-use/ https://blog.cloudflare.com/the-effect-of-switching-to-tcmal...

dilyevsky 3y ago

They better hurry up with that - i think 16.04 only getting secuirty patches until the end of this year without enterprise license

SoftTalker 3y ago
It's already past EOL unless you're paying for an "Ubuntu Pro" subscription. As is 18.04.

ceeam 3y ago

Well, Linus told you ages ago that the "fundamental design principle of Linux is to have fun". So they change stuff like this for fun. Isn't it fun for you too? And I will stick to FreeBSD, thank you. And yes, I understand that technically glibc is not a part of "Linux", but...

com2kid 3y ago

The number of arguments I've gotten into with other engineers that "manual memory allocation is deterministic, GCs are not!" is too high. First off, GCs are quite deterministic, unless they are calling rand() someplace. I've walked through GC code under controlled conditions with identical repeated allocations, GCs do the exact same thing under the exact same circumstances every time. Second off, you have to pay the price of allocation and deal with memory fragmentation somewhere. GCs typically pay the price at the tail end (deallocation, compaction) but have absurdly fast allocators (a handful of instructions), where as manual memory allocators pay the price for allocation (finding a slab of memory) up front, and unless you know how to work around the details of your allocator, you end up dealing with fragmentation yourself. FWIW this also means many, many, types of applications can gain a boost from intelligent memory allocation strategies. There is the famous example of the cruise missile that never worried about deallocation because it would explode before it ran out of memory, but scenarios involving bespoke memory management being better than "just use libc!" are actually quite common. Have a stateless microservice? If you know the max size of memory it'll use, for each connection that comes in just allocate a slab of memory and when the connection closes free the entire slab, far more efficient than using a GC or any standard allocator. Also having to calculate the max amount of memory a single connection can use is a great way to figure out how much load an instance can handle (depending if you are CPU bound or not). For certain stages of compilation, compilers can get away with not freeing memory and just exiting and letting the OS reclaim everything. (Of course less doable now days with compilers running as background services recompiling code as it is typed, and also whole program optimization has made it so compilers use a lot more memory, so I don't know if this strategy is still in use in these modern times!) The tl;dr is that freeing memory is complicated no matter if you have a GC or not. It is something every developer should be thinking about from time to time.

Karellen 3y ago
I thought that GCs are often run in their own thread, and because scheduling at the OS layer is seen as non-deterministic from the application's viewpoint (you never know when your own threads are going to get stopped/started), you can't control exactly when the GC is run? Is that wrong? Or obsolete? Or GC-dependent? But also, just because something is theoretically deterministic (e.g. it might technically be possible to work out when a GC sweep will run, and multiple runs of the exact same program will cause the exact same set of GC sweeps), actually working out when a GC sweep will run is so non-trivial the only way to actually work it out is to run the program and watch it happen, and also it becomes impossible to predict how any changes at all to a program will affect how the GC runs as a result (other than making the change and running it). Or are there ways to deduce when GC sweeps will run for a given program, without actually running it?

SilasHaslam 3y ago

I read the headline as: When alligators are boarding your precious memories. Sigh..

kelnos 3y ago

It was frustrating that the article ended with no real resolution. It was posted almost a year and a half ago; I wonder if there actually is some follow-up somewhere.