11 ms·
A new Linux memory controller promises to save lots of RAM
- ckdarby 7y ago> With this new controller, it’s possible to gain anywhere from 35-42% better memory usage in Linux. Considering the gain I'm surprised there wasn't more discussion or interest in this.
- makapuf 7y agoIt is apparently only slab memory usage, not main memory usage.
- fintler 7y agoWhat do you mean by main memory? The slab allocator is used for almost everything (e.g. page cache).
- otterley 7y agoI suspect he means anonymous pages allocated from the heap.
- sh-run 7y agoSlab would be part of main memory, not to be confused with swap. ~10% of the utilization on my system is slab. [bjames@lwks1 ~]$ grep "Slab\|MemTotal\|Active:" /proc/meminfo MemTotal: 65976620 kB Active: 6139820 kB Slab: 605988 kB That's hardly insignificant. It sounds to me like slab allocation is specifically used to allocate small blocks of memory (less than a page). Ie a slab might be set aside for integers, then if an application (edit: not applications, see rayiner's comment below) needs to store an integer it gets stored in that slab instead of somewhere else in memory. I'm not a Linux developer, so I hope I'm not spreading bad info. Maybe someone else can chime in. I assume this would largely be short lived allocations so I can see where optimizations here would lead to power savings.
- rayiner 7y agoThe slab allocator is for sub-page-size kernel objects. User space programs allocate entire virtual memory pages and use a different user-level allocator you allocate sub-page-size objects.
- yellowapple 7y agoMemory savings are still memory savings, whether they happen in kernel-space or user-space.
- coryrc 7y agoMost memory usage is not in the kernel. It's not a big effect overall.
- yellowapple 7y agoPer GGP's comment at least 605MB of memory is slab memory¹. That's around 25% (if not more) of the memory on an average smartphone. Apparently you disagree, but I'd call a reduction of that a pretty darn big effect. ¹ On my machine it's currently at only about 165MB, so not quite as extreme, but still a decent chunk of memory.
- derefr 7y agoHowever, you kind of want most memory usage to be in the kernel. Having something happen entirely in kernelspace is always better than having half of it happen in userspace. sendfile(2), for example. To the degree that you can achieve "getting the kernel to do your work for you", the kernel memory allocator becomes one of the main determinants on your scaling requirements. IIRC WhatsApp hit this point with the FreeBSD kernel, and had to tune the heck out of the kernel to keep scaling. (Tangent: why is it that we hear a lot about bespoke unikernels, and a lot about entirely-in-userspace designs like Snabb, but nobody talks about entirely-in-kernelspace designs? Linux itself makes a pretty good "unikernel framework": just write your business logic as a kernel driver (in e.g. Rust), compile it into a monolithic kernel, and then run no userland whatsoever.)
- AtlasBarfed 7y agoSo they are sharing common code segments like libraries? I hope this has some code/data awareness.
- sverige 7y agoSecurity isn't as important as performance.
- sachdevap 7y agoIs this a serious comment? I'm genuinely uncertain whether you mean this seriously.
- TheOtherDave 7y agoIt kinda depends on the context. If your machine isn’t on the internet and is dedicated to running like a large computational fluid dynamics simulation or something, security isn’t as much of an issue because there is no untrusted code running on the machine and there’s no way for hackers to access it. If your machine is Google’s public DNS server, security is critical.
- sachdevap 7y ago> If your machine isn’t on the internet I don't think there are many such machines anymore. I am not seeing this as a common case at all. And in the present context of the Linux memory controller, they would never sacrifice general kernel security for performance.
- snovv_crash 7y agoOutside of your bubble they are pretty relevant. Hell, the biggest computers in the work generally run Linux without an internet connection.
- 7y ago
- droopyEyelids 7y agoKind of interesting to think how much money/energy this can save across the planet
- TylerE 7y agoSeems like near zero. RAM is specced based on worse case, generally - and it's not like this is going to save that much anyway since it's not user-space memory.
- vbezhenar 7y agoUnused RAM usually is utilized for filesystem caches. So more cache, less disk usage, less power usage. Though difference must be truly negligible. Those trends of computing ecology terrifying me. People are talking about truly insignificant things. Yes, at large scale even 0.001% of energy saving seems like a significant number. But large scale is planetary scale and now on that scale it's again rounding error.
- TylerE 7y agoIf you want to save energy, do everything you can to kill cryptocurrency. Now THAT is some obscene waste of energy.
- imhoguy 7y agoI wonder how much impact would uninstalling social apps from smartphones have. Getting rid of FB, TW, WA clients pooling made my phone run for 2 days without charging vs 1 previously. Now at 1 billion users scale...
- xmodem 7y agoWell, the Jevons paradox would probably apply to any significant gains: https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
- dmead 7y agoit's definitely not zero. if you have a million machines in your data centers then it easily adds up to something quantifiable.
- boulos 7y agoOff-topic: it's too bad that The New Stack reproduced verbatim the term "Memory Controller" instead of "Memory Allocator" (as mm/slab.h agrees). I was really confused as to how there was going to be a Linux Memory Controller, and thought some new-fangled AMD/Intel memory controller thing had moved into software...
- derefr 7y agoMy first mental image was of an MMU that was running Linux internally—like the Xeon Phi, but even more ridiculous.
- boulos 7y agoYes, though Xeon Phi had a BSD :).
- ChickeNES 7y agoFor those of us that don't know what the above is about: https://news.ycombinator.com/item?id=12293771 https://news.ycombinator.com/item?id=12293771
- wyldfire 7y agoI thought it was an MMU designed to work with a specific Linux memory feature.
- rdc12 7y agoMy first jump was some major change to the MM subsystem
- mlyle 7y agoI've used a couple DMA controllers that are Turing complete-- it's pretty cool what kinds of things you can offload.
- nine_k 7y agoTuring completeness means undecidability of termination, not something you'd enjoy in DMA operations!
- xyzzy_plugh 7y agoI've taken a brief read through the patch series. It's a noble effort, but from the light initial feedback there will a few rounds of rework before this is a candidate in a release. This also feels unlikely to be backported, given the numerous subtle changes (e.g. cgroups v1 incompatibilities, byte vs. page accounting). This really seems like a somewhat obvious caveat of SLUB, in hindsight at least, given how memory cgroups work. The accounting overhead here seems like opening a complexity can of worms. For memory-pressure intensive short-lived applications (e.g. Hadoop jobs with no strict NUMA affinity), this could net some real benefits when processes jump across cores, let alone physical cores. Most serious folks should be setting at least some CPU affinity anyways and discourage the scheduler from bouncing processes between cores, which otherwise exasperates the issue mitigated by this patch set. If you've ever wondered why you struggle to make use of all of your RAM on a large many-core host, this is potentially a big reason why.
- akdor1154 7y ago> exasperates exacerbates? :)
- Nicci00 7y agoI wonder if these improvements will trickle down to desktop Linux users.
- doubleunplussed 7y agoBasically if it is accepted into the mainline kernel then yes, otherwise not unless people decide to install custom kernels. Distros backport a little bit, but generally don't include anything not accepted into mainline. Then there are a few popular custom kernels that include changes not included in mainline because they are not generally useful, and only make sense for specific workloads.
- washadjeffmad 7y agoLikely immediately for HEDT owners. I make use of VFIO on my home Threadripper, and while it's "only" 12 cores and 64GB RAM, it's NUMA so I have to use thread pinning to keep cores on the same die so they're not reaching across the Infinity Fabric to the other memory controllers. With better memory allocation, I could assign >12 vCPUs on performance oriented VMs or use more than half my memory without incurring a latency penalty.
- aloknnikhil 7y agoThe optimization is primarily aimed at systems with multiple cgroups, so think Docker-like container platforms.
- cbarrick 7y agoTraditional servers should be positively effected as well [1]. > Also, there is nothing fb-specific. You can take any new modern distributive (I've tried Fedora 30), boot it up and look at the amount of slab memory. Numbers are roughly the same. The explanation given in the article is that most distributions with systemd spin up a bunch of cgroups even without a container-oriented workload. [1]: https://lkml.org/lkml/2019/9/19/628 https://lkml.org/lkml/2019/9/19/628
- hliyan 7y agoThis is correct. Looks like they're proposing to share memory slabs across cgroups. But cgroups being a mechanism that's meant to partition groups of tasks for resource control, I don't know how wise it is to have them share memory slabs.
- meanderer 7y agoNot familiar with this, but I assume measures are taken so that we don’t accidentally share memory content between different processes through slabs? That seems a pretty straight forward security consideration.
- denton-scratch 7y ago"it could find its way into the mainline kernel as early as 2020" Gosh. So soon?
- progval 7y agoI'm not a kernel dev, but this looks like a pretty big change to a critical system, with very diverse non-trivial effects on performance depending on workloads. It needs to be properly reviewed and tested before landing, and that takes time.
- bcaa7f3a8bbc 7y ago> much-improved memory utilization between multiple memory cgroups There will be no improvement if I don't use memory cgroups, which means it's not related to a typical desktop or server without containers. But still good news, the use of containers can only expand.
- nicolaslem 7y agoSystemd uses cgroups heavily, so chances are your typical distribution runs hundreds of them under the hood.
- bcaa7f3a8bbc 7y agoThanks for reminding me about systemd. I just used "systemd-cgtop", and yes, I see 38 cgroups, one cgroup for each service/user. But 88% of the memory is used by one cgroup: user.slice, for all the program started by me under the desktop. So I'm not sure it can save a lot of memory on container-less desktop. Surely good for all servers.
- speedplane 7y agoThis article is so full of Linux jargon, it’s impossible to follow unless you’re deeply steeped in it. Is there another source? Memory controllers have been developed for 30+ years, I’m surprised and slightly skeptical there are still major breakthroughs, would love to read an explanation that I can actually understand.
- smcl 7y agoI randomly stumbled blindly into the world of slab allocation and wrote a little bit about it here: https://blog.mclemon.io/discover-a-linux-utility-slabtop https://blog.mclemon.io/discover-a-linux-utility-slabtop That might help at least with the slab part.
- timwaagh 7y agoWhat's the point when we can just download more RAM?