4 ms·

I have some questions. I started reading about systemd-oomd, then followed the link to its manpage (at https://www.man7.org/linux/man-pages/man8/systemd-oomd.8

by znpy 6y ago

I have some questions. I started reading about systemd-oomd, then followed the link to its manpage (at https://www.man7.org/linux/man-pages/man8/systemd-oomd.8.html https://www.man7.org/linux/man-pages/man8/systemd-oomd.8.htm...) then followed a link from there to this page: "In defence of swap: common misconceptions" at https://chrisdown.name/2018/01/02/in-defence-of-swap.html https://chrisdown.name/2018/01/02/in-defence-of-swap.html). The first line of the tl;dr is: "Having swap is a reasonably important part of a well functioning system. Without it, sane memory management becomes harder to achieve." Now, afaik linux as a kernel is pretty much designed to have swap memory and really not meant to be ran without it. Except disabling swap is required to run kubernetes, so much so that the kubelet will plain refuse to start if it detects swap memory enabled. Why?

geofft 6y ago
This is one of the primary reasons the Kubernetes folks are working on letting you run with swap! See recent discussion in https://github.com/kubernetes/kubernetes/issues/53533 https://github.com/kubernetes/kubernetes/issues/53533 . The historic reason Kubernetes didn't let you run with swap is that it's complicated from a resource management perspective - it's not clear whether each pod should be requesting some amount of swap the same way it requests some amount of RAM, and also the container runtimes didn't really have support for that (largely because you need cgroupv2 to have a hope of that sort of per-cgroup swap resource control working at all). But it's not an insurmountable problem, and it sounds from that ticket like they recognize that letting systems have swap enabled for the sake of a userspace oomd is an important use case.
xyzzy_plugh 6y ago
Why is Linux not meant to be ran without swap? In 2021? I've never heard of this. I've operated massive clusters of thousands of hosts with no swap enabled with zero hiccups. Swap comes with a lot of caveats -- when a performance-critical process begins swapping it can on occasion be more useful to just let it OOM instead. My personal desktop has 64GB of memory, with no swap enabled. Why would I want swap it I don't need more than that?
- axaxs 6y ago
  You don't need it. Linux works fine without swap, given ample memory. In my experience, even with ample swap, when Linux gets memory starved it goes haywire anyways. I'm an ex linux sysadmin, and haven't run swap on my personal systems in years.
aidenn0 6y ago
Linux works fine without swap. It also works fine with swap. It does not work fine under memory contention regardless of swap, but it is much, much worse if you have many GB of swap on a spinning disk. Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process. It used to be that the recommendation for swap was 2x ram. This recommendation was from the days were single-digit mega bytes of ram was common, and also from when linux core-dumps were both expected, and stored to swap. Fast forward to 2010ish. You might have a 64GB or 128GB SSD, but you probably also have a spinning drive. you have 8 or 16GB of ram, so your 16-32GB swap would be 1/4 the SSD, plus you heard that putting swap on an SSD wears it out faster, so you put a 16-32GB swap file on the spinning drive. Lets say you are developing software and you accidentally allocate in a tight loop. At some point the majority of the pages in ram not from the malloc loop end up in swap, so you try to do something to kill it, but Xorg and xterm both have their pages scattered across spinning drives, and your program is allocating memory faster than your machine saps in. If you are patient enough to not hard reset, an hour later, swap fills up and the OOM killer starts killing programs, and probably (but not definitely) kills the buggy malloc loop before it kills something more vital. You are annoyed, so disable swap and do the same thing: the OOM killer kills probably just your web browser (which is virtually indistinguishable to a malloc loop with Web 2.0), and the malloc loop. The system recovers in minutes. You now swear off ever running with swap again. Today: If you have a spinning disk 1GB or swap on it is fine, but zram is another good alternative. If you have slow and/or fragile SSD storage (e.g. sdcard or eMMC), zram is a good option. If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.
- kalleboo 6y ago
  > If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself Not sure about Linux, but under macOS I had a bug that was leaking uncompressed 12 MP photos in a loop. I was browsing the web while waiting for my process to run and I got up to something like 150 GB of swap used (macOS uses dynamically growing swap files) before I even noticed that there was something going on.

by znpy 6y ago

geofft 6y ago

This is one of the primary reasons the Kubernetes folks are working on letting you run with swap! See recent discussion in https://github.com/kubernetes/kubernetes/issues/53533 https://github.com/kubernetes/kubernetes/issues/53533 . The historic reason Kubernetes didn't let you run with swap is that it's complicated from a resource management perspective - it's not clear whether each pod should be requesting some amount of swap the same way it requests some amount of RAM, and also the container runtimes didn't really have support for that (largely because you need cgroupv2 to have a hope of that sort of per-cgroup swap resource control working at all). But it's not an insurmountable problem, and it sounds from that ticket like they recognize that letting systems have swap enabled for the sake of a userspace oomd is an important use case.

xyzzy_plugh 6y ago

Why is Linux not meant to be ran without swap? In 2021? I've never heard of this. I've operated massive clusters of thousands of hosts with no swap enabled with zero hiccups. Swap comes with a lot of caveats -- when a performance-critical process begins swapping it can on occasion be more useful to just let it OOM instead. My personal desktop has 64GB of memory, with no swap enabled. Why would I want swap it I don't need more than that?

axaxs 6y ago
You don't need it. Linux works fine without swap, given ample memory. In my experience, even with ample swap, when Linux gets memory starved it goes haywire anyways. I'm an ex linux sysadmin, and haven't run swap on my personal systems in years.

aidenn0 6y ago

Linux works fine without swap. It also works fine with swap. It does not work fine under memory contention regardless of swap, but it is much, much worse if you have many GB of swap on a spinning disk. Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process. It used to be that the recommendation for swap was 2x ram. This recommendation was from the days were single-digit mega bytes of ram was common, and also from when linux core-dumps were both expected, and stored to swap. Fast forward to 2010ish. You might have a 64GB or 128GB SSD, but you probably also have a spinning drive. you have 8 or 16GB of ram, so your 16-32GB swap would be 1/4 the SSD, plus you heard that putting swap on an SSD wears it out faster, so you put a 16-32GB swap file on the spinning drive. Lets say you are developing software and you accidentally allocate in a tight loop. At some point the majority of the pages in ram not from the malloc loop end up in swap, so you try to do something to kill it, but Xorg and xterm both have their pages scattered across spinning drives, and your program is allocating memory faster than your machine saps in. If you are patient enough to not hard reset, an hour later, swap fills up and the OOM killer starts killing programs, and probably (but not definitely) kills the buggy malloc loop before it kills something more vital. You are annoyed, so disable swap and do the same thing: the OOM killer kills probably just your web browser (which is virtually indistinguishable to a malloc loop with Web 2.0), and the malloc loop. The system recovers in minutes. You now swear off ever running with swap again. Today: If you have a spinning disk 1GB or swap on it is fine, but zram is another good alternative. If you have slow and/or fragile SSD storage (e.g. sdcard or eMMC), zram is a good option. If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.

kalleboo 6y ago
> If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself Not sure about Linux, but under macOS I had a bug that was leaking uncompressed 12 MP photos in a loop. I was browsing the web while waiting for my process to run and I got up to something like 150 GB of swap used (macOS uses dynamically growing swap files) before I even noticed that there was something going on.