4 ms·

Swap (her solution), xor the kernel should just OOM kill Chrome; but spinning the disk for half an hour to accomplish border-line nothing for a human that has l

Swap (her solution), xor the kernel should just OOM kill Chrome; but spinning the disk for half an hour to accomplish border-line nothing for a human that has long since gotten bored and left is pointless — the behavior noted in the LKML post is severely annoying, and not productive. My previous team was also a "no swap in prod", and this behavior bit us more than I care to admit. The devs were occasionally on the side of "swap for safety", ops was religious no-swap, and ugh. It can take 10+, 30+ minutes for systems encountering this to resolve to some meaningful conclusion, and half the time, I'm desperately trying to ssh in so-as to kill -9 the errant task anyway but ssh is paged out, and I wish the OOM killer would just do it for me instead of Linux trying to page everything through what feels like a single 4KiB page. I need to play around will sysctls more on some sort of test rig. On AWS instances with EBS disks (most instances), disk is basically network. I once suggested "cgroup'ing" (loosely speaking) the entire system into two rough buckets: one for SSH, with enough dedicated RAM that ssh will never get swapped, and one for everything else. Also, I feel like the number of devs out there who understand that mmap'd files — including binaries/libraries — are basically mini-swap files when memory pressure is high is really low; more than once I've diagnosed a machine as "page thrashing" to get back "what? but it has no swap that cannot be?". Well, pgmajfault and disk I/O metrics don't lie.

secure 7y ago
> I once suggested "cgroup'ing" (loosely speaking) the entire system into two rough buckets: one for SSH, with enough dedicated RAM that ssh will never get swapped, and one for everything else. You can use memlockd for that: https://manpages.debian.org/buster/memlockd/memlockd.8.en.html https://manpages.debian.org/buster/memlockd/memlockd.8.en.ht...
- aardvarklegend 7y ago
  Sorry, but that’s not enough. You need more ram to actually make the ssh connection useful. You can lock sshd and then be blocked from running your shell or any commands in that shell. You really do need reserved unused memory.
  - Godel_unicode 7y ago
    This approaches understanding why this isn't done; it's not just reserved memory for SSH, it's reserved memory for SSH plus bash plus the vim, top, grep, kill, etc commands you'll be running once you SSH. How much RAM does top need? What about vim? Now realize you're imposing this penalty on every embedded, low-memory Linux device which might not ever have this problem.
    - Dylan16807 7y ago
      But for the actual goal you don't have to reserve it as empty, you just have to prioritize it.
    - pixl97 7y ago
      Why would you be imposing this on every device when I would assume it would be controlled by a settings file in /etc
    - kevin_thibedeau 7y ago
      Linux was quite usable in 4MiB 25 years ago. Just give yourself a minimal busybox environment for rescue activities.
isostatic 7y ago
I don't have problems with my desktop (16G ram no swap). I do run into OOM on some servers (I've just migrated a service from a 512MB instance to a 2G one because it crashed every few months) When I have had swap, the machine simply hangs rather than OOM saving the machine. What I don't understand is how 500M of swap can help on a 16G machine. Why would it be any better than an 18G machine?
- Diggsey 7y ago
  Maybe if the kernel behaved optimally it wouldn't be any different, but empirically it seems like it does help. My guess would be that if it's all RAM, then the kernel will happily use all of it for IO buffers, etc. and make no attempt to reduce its usage until actually necessary. If some of it is swap, then the kernel will (over time) try to reduce the size of its IO buffers until swap is no longer needed, but that de-allocation doesn't need to happen immediately, or block the rest of the system whilst it does happen. Maybe there is a way to have that "high water mark" without needing swap - reserve 500M to be only used in "emergencies" - I don't know the details of the linux kernel well enough to know if that's a possibility.
  - tomsmeding 7y ago
    Now I'm suddenly wondering if this isn't really easy after all (or rather: should be really easy if all was well). If Linux behaves better when it has some slack space past the normal RAM that it doesn't like to put stuff in but will when necessary (i.e. swap), let it do the exact same thing to the last portion of your RAM. In other words: make a swap file of, say, 10% of your memory size, and put it in a tmpfs. Linux probably doesn't want that I believe, but it would, if working, give the results you want.
    - Dylan16807 7y ago
      > make a swap file of, say, 10% of your memory size, and put it in a tmpfs You're looking for zram, I believe.
- toast0 7y ago
saurik 7y ago
> I once suggested "cgroup'ing" (loosely speaking) the entire system into two rough buckets: one for SSH, with enough dedicated RAM that ssh will never get swapped, and one for everything else. I feel like I've been waiting 25 years for someone to implement this by default in every operating system, and yet no one seems to do this. (Arguably iOS does this with Jetsam priorities, btw.) Why is it so hard to ensure that these base components get guaranteed RAM? It was always extra infurating on Windows, as you'd end up in swap hell and need to kill something because the computer is now running insanely slow; so you hit ctrl-alt-del and the UI would instantly respond with a menu that would work great... and then you'd click Task Manager, and rather than that functionality being implemented in that menu--or being designed to never swap--you would get thrown back into the swap hell in the hope that eventually the taskman would load. It was insane, and so easily fixed in numerous ways (the most simple probably being to add a trivial task manager that only had memory usage and a kill button in that OS escape menu).
- nailer 7y ago
  > I feel like I've been waiting 25 years for someone to implement this by default in every operating system, and yet no one seems to do this. Damn straight. The xbox builds of Windows do this too (so the console is still controllable when a game is running) and I swear the lack of control in desktop OSs (can't start Task Manager to kill the thing that's stopping you from starting anything) will kill desktop computing if mobile OSs increase their capabilities. A device that isn't responsive to input is indistinguishable from the device being broken.
- erentz 7y ago
  This principal of reserving resources for OAM is also something people should really think about in the network too. Provide some dedicated resources for OAM traffic. I’ve seen too many people be anti-QoS for no real reason (it’s too complicated for them, or surprisingly common is they believe QoS is for “dumb enterprises” or for “legacy telco service providers not us cool new guys”). This includes one very large well known cloud provider that still resisted this even after traffic from multiple DOS events interrupted their monitoring and management access.
- a_t48 7y ago
VectorLock 7y ago
>It can take 10+, 30+ minutes for systems encountering this to resolve to some meaningful conclusion, and half the time, I'm desperately trying to ssh in so-as to kill -9 the errant task anyway but ssh is paged out, and I wish the OOM killer would just do it for me instead of Linux trying to page everything through what feels like a single 4KiB page. Sounds like you have a redundancy problem not a swap problem. If should just be able to kill a machine that gets into a bad way like that and move on. What if it wasn't swap but one of the million other things that could make your server crawl?
- deathanatos 7y ago
  Generally speaking, swap/page thrashing is very easy to pick out from the metrics on a VM. (We use a system called Prometheus[1], which records and transmits metrics to a certralized service.) In particular, a machine that is swap/page thrashing will generally show as having no available RAM, a high amount (especially relative to baseline) of major (required disk reads) page faults, and often the CPU profile will be spending a lot of time in I/O wait, too, I think, though I usually just use out of RAM + page faulting. Also, the metrics service tends to go dark shortly afterwards — it's having the same issue as everything else on the VM at getting CPU time. Major page faults, perhaps the key stat for "this machine is page thrashing" since it directly corresponds to it, is found in /proc/vmstat and is called "pgmajfault". Though like I said, we generally had Prometheus and Grafana to turn these into pretty graphs, and to export them out of the VM itself, since when something is page thrashing, getting it to do anything is hard. CPU contention lacks the "out of RAM" part, and won't knock the metrics offline. Network contention can knock out the metrics, but often doesn't, and lacks the other signals: out of RAM/page faults. Disk I/O lacks the out of RAM & doesn't knock the metrics out since they don't require (beyond being paged in) disk I/O. (And those — CPU, RAM, network, and disk-ish — are about the only real resource dimensions on a VM.) Alternatively, if you let it play out and the VM eventually recovers, it might nonetheless decide to OOM kill a thing or to along the way, and those show up in dmesg / on the console, in you can get to those. > If should just be able to kill a machine that gets into a bad way like that and move on. I must admit that the production I love and cared for was not always perfect. Often, yes, I could, but there were a few spots where things weren't so rosy. Even when I could, I generally wanted to have some understanding as to why the VM went under, so as to not have the problem come back again, later, on a different VM. Page thrashing, in particular, is basically always symptomatic of a bug. And in distributed systems, the bugs are also distributed. The economics of getting devs enough time to develop to that quality vs. management wanting new features has been one of the hardest challenges of my career. Also for a while I lacked permission to actually kill a machine because ops was locking permissions down because "devs shouldn't have access to the actual machines"; like she says in one of the linked posts, fine, have my pager, you deal with the pages. [1]: https://prometheus.io/ https://prometheus.io/
nerpderp82 7y ago
We had a Hadoop cluster that ran noswap, it killed our ability to use all the ram and when it did exhaust ram, which was all the time, it would often KP the whole machine. No swap is not the fix. I love the idea of protecting core components, maybe even give them a core or two and dedicated ram.
- pacala 7y ago
  With all due respect, this poor design on part of Hadoop and/or Hadoop configuration. Hadoop workers should be bounded to certain amount hardware resources, and killed if they surpass them. As opposed to killing random processes / kernel panic / etc. On Linux, this is realised via cgroups, which originated in Borg and today is widely used by Kubernetes.
  - GauntletWizard 7y ago
    I am not-incredibly-surprised that none of these many talks about swap have mentioned kubernetes. Kubernetes does not solve this problem conclusively, but it does solve it to a huge extent. Your Application Cgroups have lower RAM priority than system tasks. You allocate enough RAM for system-level tasks in kubernetes ( --system-reserved and --kube-reserved). You run your application tasks in a fixed reservation, and any that are over their reservation are first to be killed. If application tasks are repeatedly getting killed, your node is faulty and gets restarted. All of this is done with little fuss. At some higher level, faulty processes are easy to identify because they are the most evicted (And if your system-level daemons are what's at fault, it gets really obvious because of how unobvious the application is) https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ https://kubernetes.io/docs/tasks/administer-cluster/reserve-...
  - nerpderp82 7y ago
    I was only "a dev" and had an extremely difficult time getting cluster configuration changes pushed through. It was only because I bribed a sysop that we got a swap file added. We had our own Mr. No to deal with. Lots of things could/should and would be done differently.
zrm 7y ago
> Also, I feel like the number of devs out there who understand that mmap'd files — including binaries/libraries — are basically mini-swap files when memory pressure is high is really low; more than once I've diagnosed a machine as "page thrashing" to get back "what? but it has no swap that cannot be?". This is also one of the reasons that putting swap on a different physical drive than /usr and /bin can help a lot. Once you hit memory pressure, I/O caching is going to disappear at the same time as your I/O is saturated doing swap. Just being able to read /usr/bin/ssh from a different drive than the one swap is thrashing can be night and day.
paulddraper 7y ago
One thing many people don't realize is that Linux OOM killer is nearly broken in a lot of cases. For example, if you don't have swap space. Without swap, Linux is winds up flushing file caches, mmaped files, etc. and usually re-reading them all over again...the machine is technically not out of memory or "swapping", but effectively, it's swapping and unresponsive.

secure 7y ago

> I once suggested "cgroup'ing" (loosely speaking) the entire system into two rough buckets: one for SSH, with enough dedicated RAM that ssh will never get swapped, and one for everything else. You can use memlockd for that: https://manpages.debian.org/buster/memlockd/memlockd.8.en.html https://manpages.debian.org/buster/memlockd/memlockd.8.en.ht...

aardvarklegend 7y ago
Sorry, but that’s not enough. You need more ram to actually make the ssh connection useful. You can lock sshd and then be blocked from running your shell or any commands in that shell. You really do need reserved unused memory.
- Godel_unicode 7y ago
  This approaches understanding why this isn't done; it's not just reserved memory for SSH, it's reserved memory for SSH plus bash plus the vim, top, grep, kill, etc commands you'll be running once you SSH. How much RAM does top need? What about vim? Now realize you're imposing this penalty on every embedded, low-memory Linux device which might not ever have this problem.
  - Dylan16807 7y ago
    But for the actual goal you don't have to reserve it as empty, you just have to prioritize it.
  - pixl97 7y ago
    Why would you be imposing this on every device when I would assume it would be controlled by a settings file in /etc
  - kevin_thibedeau 7y ago
    Linux was quite usable in 4MiB 25 years ago. Just give yourself a minimal busybox environment for rescue activities.

isostatic 7y ago

I don't have problems with my desktop (16G ram no swap). I do run into OOM on some servers (I've just migrated a service from a 512MB instance to a 2G one because it crashed every few months) When I have had swap, the machine simply hangs rather than OOM saving the machine. What I don't understand is how 500M of swap can help on a 16G machine. Why would it be any better than an 18G machine?

Diggsey 7y ago
Maybe if the kernel behaved optimally it wouldn't be any different, but empirically it seems like it does help. My guess would be that if it's all RAM, then the kernel will happily use all of it for IO buffers, etc. and make no attempt to reduce its usage until actually necessary. If some of it is swap, then the kernel will (over time) try to reduce the size of its IO buffers until swap is no longer needed, but that de-allocation doesn't need to happen immediately, or block the rest of the system whilst it does happen. Maybe there is a way to have that "high water mark" without needing swap - reserve 500M to be only used in "emergencies" - I don't know the details of the linux kernel well enough to know if that's a possibility.
- tomsmeding 7y ago
  Now I'm suddenly wondering if this isn't really easy after all (or rather: should be really easy if all was well). If Linux behaves better when it has some slack space past the normal RAM that it doesn't like to put stuff in but will when necessary (i.e. swap), let it do the exact same thing to the last portion of your RAM. In other words: make a swap file of, say, 10% of your memory size, and put it in a tmpfs. Linux probably doesn't want that I believe, but it would, if working, give the results you want.
  - Dylan16807 7y ago
    > make a swap file of, say, 10% of your memory size, and put it in a tmpfs You're looking for zram, I believe.
toast0 7y ago

saurik 7y ago

> I once suggested "cgroup'ing" (loosely speaking) the entire system into two rough buckets: one for SSH, with enough dedicated RAM that ssh will never get swapped, and one for everything else. I feel like I've been waiting 25 years for someone to implement this by default in every operating system, and yet no one seems to do this. (Arguably iOS does this with Jetsam priorities, btw.) Why is it so hard to ensure that these base components get guaranteed RAM? It was always extra infurating on Windows, as you'd end up in swap hell and need to kill something because the computer is now running insanely slow; so you hit ctrl-alt-del and the UI would instantly respond with a menu that would work great... and then you'd click Task Manager, and rather than that functionality being implemented in that menu--or being designed to never swap--you would get thrown back into the swap hell in the hope that eventually the taskman would load. It was insane, and so easily fixed in numerous ways (the most simple probably being to add a trivial task manager that only had memory usage and a kill button in that OS escape menu).

nailer 7y ago
> I feel like I've been waiting 25 years for someone to implement this by default in every operating system, and yet no one seems to do this. Damn straight. The xbox builds of Windows do this too (so the console is still controllable when a game is running) and I swear the lack of control in desktop OSs (can't start Task Manager to kill the thing that's stopping you from starting anything) will kill desktop computing if mobile OSs increase their capabilities. A device that isn't responsive to input is indistinguishable from the device being broken.
erentz 7y ago
This principal of reserving resources for OAM is also something people should really think about in the network too. Provide some dedicated resources for OAM traffic. I’ve seen too many people be anti-QoS for no real reason (it’s too complicated for them, or surprisingly common is they believe QoS is for “dumb enterprises” or for “legacy telco service providers not us cool new guys”). This includes one very large well known cloud provider that still resisted this even after traffic from multiple DOS events interrupted their monitoring and management access.
a_t48 7y ago

VectorLock 7y ago

>It can take 10+, 30+ minutes for systems encountering this to resolve to some meaningful conclusion, and half the time, I'm desperately trying to ssh in so-as to kill -9 the errant task anyway but ssh is paged out, and I wish the OOM killer would just do it for me instead of Linux trying to page everything through what feels like a single 4KiB page. Sounds like you have a redundancy problem not a swap problem. If should just be able to kill a machine that gets into a bad way like that and move on. What if it wasn't swap but one of the million other things that could make your server crawl?

deathanatos 7y ago
Generally speaking, swap/page thrashing is very easy to pick out from the metrics on a VM. (We use a system called Prometheus[1], which records and transmits metrics to a certralized service.) In particular, a machine that is swap/page thrashing will generally show as having no available RAM, a high amount (especially relative to baseline) of major (required disk reads) page faults, and often the CPU profile will be spending a lot of time in I/O wait, too, I think, though I usually just use out of RAM + page faulting. Also, the metrics service tends to go dark shortly afterwards — it's having the same issue as everything else on the VM at getting CPU time. Major page faults, perhaps the key stat for "this machine is page thrashing" since it directly corresponds to it, is found in /proc/vmstat and is called "pgmajfault". Though like I said, we generally had Prometheus and Grafana to turn these into pretty graphs, and to export them out of the VM itself, since when something is page thrashing, getting it to do anything is hard. CPU contention lacks the "out of RAM" part, and won't knock the metrics offline. Network contention can knock out the metrics, but often doesn't, and lacks the other signals: out of RAM/page faults. Disk I/O lacks the out of RAM & doesn't knock the metrics out since they don't require (beyond being paged in) disk I/O. (And those — CPU, RAM, network, and disk-ish — are about the only real resource dimensions on a VM.) Alternatively, if you let it play out and the VM eventually recovers, it might nonetheless decide to OOM kill a thing or to along the way, and those show up in dmesg / on the console, in you can get to those. > If should just be able to kill a machine that gets into a bad way like that and move on. I must admit that the production I love and cared for was not always perfect. Often, yes, I could, but there were a few spots where things weren't so rosy. Even when I could, I generally wanted to have some understanding as to why the VM went under, so as to not have the problem come back again, later, on a different VM. Page thrashing, in particular, is basically always symptomatic of a bug. And in distributed systems, the bugs are also distributed. The economics of getting devs enough time to develop to that quality vs. management wanting new features has been one of the hardest challenges of my career. Also for a while I lacked permission to actually kill a machine because ops was locking permissions down because "devs shouldn't have access to the actual machines"; like she says in one of the linked posts, fine, have my pager, you deal with the pages. [1]: https://prometheus.io/ https://prometheus.io/

nerpderp82 7y ago

We had a Hadoop cluster that ran noswap, it killed our ability to use all the ram and when it did exhaust ram, which was all the time, it would often KP the whole machine. No swap is not the fix. I love the idea of protecting core components, maybe even give them a core or two and dedicated ram.

pacala 7y ago
With all due respect, this poor design on part of Hadoop and/or Hadoop configuration. Hadoop workers should be bounded to certain amount hardware resources, and killed if they surpass them. As opposed to killing random processes / kernel panic / etc. On Linux, this is realised via cgroups, which originated in Borg and today is widely used by Kubernetes.
- GauntletWizard 7y ago
  I am not-incredibly-surprised that none of these many talks about swap have mentioned kubernetes. Kubernetes does not solve this problem conclusively, but it does solve it to a huge extent. Your Application Cgroups have lower RAM priority than system tasks. You allocate enough RAM for system-level tasks in kubernetes ( --system-reserved and --kube-reserved). You run your application tasks in a fixed reservation, and any that are over their reservation are first to be killed. If application tasks are repeatedly getting killed, your node is faulty and gets restarted. All of this is done with little fuss. At some higher level, faulty processes are easy to identify because they are the most evicted (And if your system-level daemons are what's at fault, it gets really obvious because of how unobvious the application is) https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ https://kubernetes.io/docs/tasks/administer-cluster/reserve-...
- nerpderp82 7y ago
  I was only "a dev" and had an extremely difficult time getting cluster configuration changes pushed through. It was only because I bribed a sysop that we got a swap file added. We had our own Mr. No to deal with. Lots of things could/should and would be done differently.

zrm 7y ago

> Also, I feel like the number of devs out there who understand that mmap'd files — including binaries/libraries — are basically mini-swap files when memory pressure is high is really low; more than once I've diagnosed a machine as "page thrashing" to get back "what? but it has no swap that cannot be?". This is also one of the reasons that putting swap on a different physical drive than /usr and /bin can help a lot. Once you hit memory pressure, I/O caching is going to disappear at the same time as your I/O is saturated doing swap. Just being able to read /usr/bin/ssh from a different drive than the one swap is thrashing can be night and day.

paulddraper 7y ago

One thing many people don't realize is that Linux OOM killer is nearly broken in a lot of cases. For example, if you don't have swap space. Without swap, Linux is winds up flushing file caches, mmaped files, etc. and usually re-reading them all over again...the machine is technically not out of memory or "swapping", but effectively, it's swapping and unresponsive.