5 ms·

A mini version of this was presented in the wonderful & amazing "Linux Memory Management At Scale" video, also by Chris Down: https://www.youtube.com/watch?v=cS

A mini version of this was presented in the wonderful & amazing "Linux Memory Management At Scale" video, also by Chris Down: https://www.youtube.com/watch?v=cSJFLBJusVY&t=11m42s https://www.youtube.com/watch?v=cSJFLBJusVY&t=11m42s You should have swap. You should have swap!! This video is one of the most amazing, useful, gem-after-gem explanations of Linux you could possibly ever encounter. There are so so few who have the in depth knowledge, across so many broad experiences, at such scale, to understand Linux's memory management. Side topic #1: this video's advice on cgroups just makes me sad, because I love Kubernetes, but Kubernetes building it's own scheduler, in contest with not in tandem with the kernel's cgroup system, is super super super f-ing sad. Kubernetes running all it's shit in a single cgroup is some decrepit sad sad shit. I still love & user Kubernetes, but god damn, for what basically amounts to a cloud os, it's using the underlying host-OS extremely extremely poorly. This video was key in making me think & consider how much we've lost, how sad it is we don't take advantage of what the kernel is trying to do for us with trading off workloads. Leaving the kubernetes scheduler to try to fit all the workloads feels like we're letting a bunch of ignorant elementary schoolers run the show. It's embarrassingly superficial. Side topic #2: it would be really interesting to try to go a bit deeper on the topic. I'd love a world where we can easily ask- what are these anonymous rarely used memory pages we get to swap out? That's the main advantage, under low & medium pressures: the system has some flexibility to move out things not really used, where-as it has to keep everything resident without swap. But answering: what are these things not being used: that probably requires bptrace foo to figure out what regions are getting swapped, plus deep skills to figure out what those addresses correspond to. I doubt Linux will just tell us, oh, I evicted this text block from the binary. (It likely doesn't know much about what blocks it's evicting.)

marcosdumay 5y ago
Keep in mind that positive memory pressure is something that if done by more than a single application at a time, will completely break your system, and if done only one application at a time will severely reduce your performance predictability (but can indeed increase throughput). So, what he talks about on that video is a very specialized performance optimization that may work on some (certainly not all) systems that you have complete control of. It isn't general advice in any way.
theamk 5y ago
Depends? If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on. If you have a latency-critical server, don't use swap. With swap, some data will get swapped out, and you will have sudden latency trying to access it. Without swap, it may still swap out the code, but the code is usually much smaller than the data, plus it is all pretty hot. If occasionally you have to process a very large datasets, and you are willing to wait a minute or twenty, you may enable swap. But be patient, and make sure your ssh has no timeout which can be hit. If you have a small amount of RAM, enable swap. It would be slow, but at least you would be able to do something without crashing.
- Teletio 5y ago
  There are CICD builds out there which consume much more resources and time were just killing one part of the build would destroy the work of hours. Not sure why you wouldn't want swap for it? It will allow you to fine-tune the build later and give that build a realistic chance to finish
  - theamk 5y ago
    Because once swap activates, build now takes hours instead of tens of minutes. So it would timeout anyway, but only after wasting lots of resources. And even if you increase the timeout a lot instead, your machine how has a bunch of things swapped out, so now your tests timeout, which is even worse. Yes, killing that part of the build did destroy the work of hours. It was still better to disable the swap than try to "ride it out".
    - Teletio 5y ago
      Things don't just take hours longer just because the Linux kernel throws out a few pages which haven't been used for a while. And it also totally depends on how much memory is missing. I still prefer to have something taking 20 minutes longer instead of failing and fine-tuning the resources after it.
jmillikin 5y ago
I don't understand your paragraph on Kubernetes. Say you have 1000 machines with 4 TiB each, and you want to schedule processes that will cumulatively allocate 4 EiB. How would you do this without letting Kubernetes handle the scheduling? Once Kubernetes decides which processes should run on each machine, it allocates separate cgroups to each process group, which it uses for enforcing CPU and RAM resource limits. You can see this by examining the cgroupfs control filesystem. How would you propose improving this?
bboreham 5y ago
I’m a little confused by your #1; I was under the impression that each container had its own cgroup. Note that Kubernetes now has code to work with swap, although some work remains. Tracking issue: https://github.com/kubernetes/enhancements/issues/2400 https://github.com/kubernetes/enhancements/issues/2400 There is a lengthy conversation in this issue, with some deep insights (and a lot of misconceptions). https://github.com/kubernetes/kubernetes/issues/53533 https://github.com/kubernetes/kubernetes/issues/53533

marcosdumay 5y ago

Keep in mind that positive memory pressure is something that if done by more than a single application at a time, will completely break your system, and if done only one application at a time will severely reduce your performance predictability (but can indeed increase throughput). So, what he talks about on that video is a very specialized performance optimization that may work on some (certainly not all) systems that you have complete control of. It isn't general advice in any way.

theamk 5y ago

Depends? If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on. If you have a latency-critical server, don't use swap. With swap, some data will get swapped out, and you will have sudden latency trying to access it. Without swap, it may still swap out the code, but the code is usually much smaller than the data, plus it is all pretty hot. If occasionally you have to process a very large datasets, and you are willing to wait a minute or twenty, you may enable swap. But be patient, and make sure your ssh has no timeout which can be hit. If you have a small amount of RAM, enable swap. It would be slow, but at least you would be able to do something without crashing.

Teletio 5y ago
There are CICD builds out there which consume much more resources and time were just killing one part of the build would destroy the work of hours. Not sure why you wouldn't want swap for it? It will allow you to fine-tune the build later and give that build a realistic chance to finish
- theamk 5y ago
  Because once swap activates, build now takes hours instead of tens of minutes. So it would timeout anyway, but only after wasting lots of resources. And even if you increase the timeout a lot instead, your machine how has a bunch of things swapped out, so now your tests timeout, which is even worse. Yes, killing that part of the build did destroy the work of hours. It was still better to disable the swap than try to "ride it out".
  - Teletio 5y ago
    Things don't just take hours longer just because the Linux kernel throws out a few pages which haven't been used for a while. And it also totally depends on how much memory is missing. I still prefer to have something taking 20 minutes longer instead of failing and fine-tuning the resources after it.

jmillikin 5y ago

I don't understand your paragraph on Kubernetes. Say you have 1000 machines with 4 TiB each, and you want to schedule processes that will cumulatively allocate 4 EiB. How would you do this without letting Kubernetes handle the scheduling? Once Kubernetes decides which processes should run on each machine, it allocates separate cgroups to each process group, which it uses for enforcing CPU and RAM resource limits. You can see this by examining the cgroupfs control filesystem. How would you propose improving this?

bboreham 5y ago

I’m a little confused by your #1; I was under the impression that each container had its own cgroup. Note that Kubernetes now has code to work with swap, although some work remains. Tracking issue: https://github.com/kubernetes/enhancements/issues/2400 https://github.com/kubernetes/enhancements/issues/2400 There is a lengthy conversation in this issue, with some deep insights (and a lot of misconceptions). https://github.com/kubernetes/kubernetes/issues/53533 https://github.com/kubernetes/kubernetes/issues/53533