9 ms·

Looking at the htop screenshot, I notice the lack of swap. You may want to enable earlyoom, so your whole server doesn't go down when a service goes bananas. Th

Looking at the htop screenshot, I notice the lack of swap. You may want to enable earlyoom, so your whole server doesn't go down when a service goes bananas. The Linux Kernel OOM killer is often a bit too late to trigger. You can also enable zram to compress ram, so you can over-provision like the pros'. A lot of long-running software leaks memory that compresses pretty well. Here is how I do it on my Hetzner bare-metal servers using Ansible: https://gist.github.com/fungiboletus/794a265cc186e79cd5eb2fe896f05168 https://gist.github.com/fungiboletus/794a265cc186e79cd5eb2fe... It also works on VMs.

cactusplant7374 1y ago
What's the performance hit from compressing ram?
- speedgoose 1y ago
  I haven’t scientifically measured, but you don’t compress the whole ram. It is more about reserving a part of the ram to have very fast swap. For an algorithm using the whole memory, that’s a terrible idea.
  - sokoloff 1y ago
    > It is more about reserving a part of the ram to have very fast swap. I understand all of those words, but none of the meaning. Why would I reserve RAM in order to put fast swap on it?
    - vlovich123 1y ago
      Swap to disk involves a relatively small pipe (usually 10x smaller than RAM). So instead of paying the cost to page out to disk immediately, you create compressed pages and store that in a dedicated RAM region for compressed swap. This has a number of benefits: in practice more “active” space is freed up as unused pages are compressed and often compressible. Often times that can be freed application memory that is reserved within application space but in the free space of the allocator, especially if that allocator zeroes it those pages in the background, but even active application memory (eg if you have a browser a lot of the memory is probably duplicated many times across processes). So for a usually invisible cost you free up more system RAM. Additionally, the overhead of the swap is typically not much more than a memcpy even compressed which means that you get dedup and if you compressed erroneously (data still needed) paging it back in is relatively cheap. It also plays really well with disk swap since the least frequently used pages of that compressed swap can be flushed to disk leaving more space in the compressed RAM region for additional pages. And since you’re flushing retrieving compressed pages from disk you’re reducing writes on an SSD (longevity) and reducing read/write volume (less overhead than naiive direct swap to disk). Basically if you think of it as tiered memory, you’ve got registers, l1 cache, l2 cache, l3 cache, normal RAM, compressed swap RAM, disk swap - it’s an extra interim tier that makes the system more efficient.
  - LargoLasskhyfv 1y ago
    >...but you don’t compress the whole ram. I do: https://postimg.cc/G8Gcp3zb https://postimg.cc/G8Gcp3zb (casualmeasurement.png)
- YouAreWRONGtoo 1y ago
  It's sometimes not a hit, because CPUs have caches and memory bandwidth is the limiting factor.
- waynesonfire 1y ago
  > zram, formerly called compcache, is a Linux kernel module for creating a compressed block device in RAM, i.e. a RAM disk with on-the-fly disk compression. The block device created with zram can then be used for swap or as a general-purpose RAM disk To clarify OP's represention of the tool, it compresses swap space not resident ram. Outside of niche use-cases, compressing swap has overall little utility.
  - coppsilgold 1y ago
    Incorrect, with zram you swap ram to compressed ram. It has the benefit of absorbing memory leaks (which for whatever reason compress really well) and compressing stale memory pages. Under actual memory pressure performance will degrade. But in many circumstances where your powerful CPU is not fully utilized you can 2x or even 3x your effective RAM (you can opt for zstd compression). zram also enables you to make the trade-off of picking a more powerful CPU for the express purpose of multiplying your RAM if the workload is compatible with the idea. PS: On laptops/workstations, zram will not interfere with an SSD swap partition if you need it for hibernation. Though it will almost never be used for anything else if you configure your zram to be 2x your system memory.
    - masklinn 1y ago
      > Incorrect, with zram you swap ram to compressed ram. That reads like what they said? You reserve part of the RAM as a swap device, and memory is swapped from resident RAM to the swap ramdisk, as long as there’s space on there. And AFAIK linux will not move pages between swap devices because it doesn’t understand them beyond priority. Zswap actually seems strictly better in many cases (especially interactive computers / dev machines) as it can more flexibly grow / shrink, and can move pages between the compressed RAM cache and the disk swap.
- aidenn0 1y ago
  Depends on the algorithm (and how much CPU is in use); if you have a spare CPU, the faster algorithms can more-or-less keep up with your memory bandwidth, making the overhead negligible. And of course the overhead is zero when you don't page-out to swap.
levkk 1y ago
Yeah, no way. As soon as you hit swap, _most_ apps are going to have a bad, bad time. This is well known, so much so that all EC2 instances in AWS disable it by default. Sure, they want to sell you more RAM, but it's also just true that swap doesn't work for today's expectations. Maybe back in the 90s, it was okay to wait 2-3 seconds for a button click, but today we just assume the thing is dead and reboot.
- gchamonlive 1y ago
  How programs use ram also changed from the 90s. Back then they were written targeting machines that they knew would have a hard time fitting all their data in memory, so hitting swap wouldn't hurt perceived performance too drastically since many operations were already optimized to balance data load between memory and disk. Nowadays when a program hits swap it's not going to fallback to a different memory usage profile that prioritises disk access. It's going to use swap as if it were actual ram, so you get to see the program choking the entire system.
  - winrid 1y ago
    Exactly. Nowadays, most web services are run in a GC'ed runtime. That VM will walk pointers all over the place and reach into swap all the time.
    - cogman10 1y ago
      Depends entirely on the runtime. If your GC is a moving collector, then absolutely this is something to watch out for. There are, however, a number of runtimes that will leave memory in place. They are effectively just calling `malloc` for the objects and `free` when the GC algorithm detects an object is dead. Go, the CLR, Ruby, Python, Swift, and I think node(?) all fit in this category. The JVM has a moving collector.
      - zozbot234 1y ago
        Every garbage collector has to constantly sift through the entire reference graph of the running program to figure out what objects have become garbage. Generational GC's can trace through the oldest generations less often, but that's about it. Tracing garbage collectors solve a single problem really really well - managing a complex, possibly cyclical reference graph, which is in fact inherent to some problems where GC is thus irreplaceable - and are just about terrible wrt. any other system-level or performance-related factor of evaluation.
shrubble 1y ago
It's always a good idea to have a tiny amount of swap just in case. Like 1GB.
- akerl_ 1y ago
  Why?
  - angch 1y ago
    Like a highway brake failure ramp, you have room for handling failures gentler. So services don't just get outright killed. If you monitor your swap usage, any usage of swap gives you early warning that your services require more memory already. Gives you some time to upgrade, or tune services before it goes ka-boom.
    - akerl_ 1y ago
      If your memory usage is creeping up, the way you'll find out that you need more memory is by monitoring memory usage via the same mechanisms you'd hypothetically use to monitor your swap usage. If your memory usage spikes suddenly, a nominal amount of swap isn't stopping anything from getting killed; you're at best buying yourself a few seconds, so unless you spend your time just staring at the server, it'll be dead anyways.
  - CGamesPlay 1y ago
    Because some portion of the RAM used by your daemons isn't actually being accessed, and using that RAM to store file cache is actually a better use than storing idle memory. The old rule about "as much swap as main memory" definitely doesn't hold any more, but a few GB to store unneeded wired memory to dedicate more room to file cache is still useful. As a small example from a default Ubuntu installation, "unattended-upgrades" is holding 22MB of RSS, and will not impact system performance at all if it spends next week swapped out. Bigger examples can be found in monolithic services where you don't use some of the features but still have to wire them into RAM. You can page those inactive sections of the individual process into swap, and never notice.
awesome_dude 1y ago
How do you get swap on a VPS?
- justsomehnguy 1y ago
  Search "linux enable swap in a file" To enable a swap file in Linux, first create the swap file using a command like sudo dd if=/dev/zero of=/swapfile bs=1G count=1 for a 1GB file. Then, set it up with sudo mkswap /swapfile and activate it using sudo swapon /swapfile. To make it permanent, add /swapfile swap swap defaults 0 0 to your /etc/fstab file.
  - awesome_dude 1y ago
    Strongly suggest you try doing that on a VPS, then report back
    - ahepp 1y ago
      What do you think is going to happen? I tested it out on an ec2 instance just now and it seems to have worked as one would expect.
      - awesome_dude 1y ago
        EC2 != VPS
        cmpxchg8b 1y ago
        They both offer virtualized guests under a hypervisor host. EC2 does have more offload specialization hardware but for the most part they are functionally equivalent, unless I'm missing something...
        ahepp 1y ago
        What VPS do you think this would cause problems on? Why?
      - 1y ago
Bender 1y ago
Another option would be to have more memory that required over-engineer and to adjust the oom score per app, adding early kill weight to non critical apps and negative weight to important apps. oom_score_adj is already set to -1000 by OpenSSH for example. NSDJUST=$(pgrep -x nsd); echo -en '-378' > /proc/"${NSDJUST}"/oom_score_adj Another useful thing to do is effecively disable over-commit on all staging and production servers (0 ratio instead of 2 memory to fully disable as these do different things, memory 0 still uses formula) vm.overcommit_memory = 0 vm.overcommit_ratio = 0 Also use a formula to set min_free and reserved memory using a formula from Redhat that I do not have handy based on installed memory. min_free can vary from 512KB to 16GB depending on installed memory. vm.admin_reserve_kbytes = 262144 vm.user_reserve_kbytes = 262144 vm.min_free_kbytes = 1024000 At least that worked for me in about 50,000 physical servers for over a decade that were not permitted to have swap and installed memory varied from 144GB to 4TB of RAM. OOM would only occur when the people configuring and pushing code would massively over-commit and not account for memory required by the kernel. Not following best practices defined by Java and thats a much longer story. Another option is to limit memory per application in cgroups but that requires more explaining than I am putting in an HN comment. Another useful thing is to never OOM kill in the first place on servers that are only doing things in memory and need not commit anything to disk. So don't do this on a disked database. This is for ephemeral nodes that should self heal. Wait 60 seconds so drac/ilo can capture crash message and then earth shattering kaboom... # cattle vs kittens, mooooo... kernel.panic = 60 vm.panic_on_oom = 2 For a funny side note, those options can also be used as a holy hand grenade to intentionally unsafely reboot NFS diskless farms when failing over to entirely different NFS server clusters. setting panic to 15 mins, triggering OOM panic by setting min_free to 16TB at the command line via Ansible not in sysctl.conf, swapping clusters, arp storm and reconverge.
- liqilin1567 1y ago
  Thanks for sharing I think these are very useful suggestions.
- benterix 1y ago
  The lengths people will go to avoid k8s... (very easy on Hetzner Cloud BTW).
statictype 1y ago
Thanks for this. We resorted to setting ram thresholds in systemd. Is earlyoom a better solution than that to prevent an erratic process from making an instance unresposnsive?
dboreham 1y ago
Haven't used swap since 2010.
TheDong 1y ago
Even better than earlyoom is systemd-oomd[0] or oomd[1]. systemd-oomd and oomd use the kernel's PSI[2] information which makes them more efficient and responsive, while earlyoom is just polling. earlyoom keeps getting suggested, even though we have PSI now, just because people are used to using it and recommending it from back before the kernel had cgroups v2. [0]: https://www.freedesktop.org/software/systemd/man/latest/systemd-oomd.service.html https://www.freedesktop.org/software/systemd/man/latest/syst... [1]: https://github.com/facebookincubator/oomd https://github.com/facebookincubator/oomd [2]: https://docs.kernel.org/accounting/psi.html https://docs.kernel.org/accounting/psi.html
- CGamesPlay 1y ago
  "earlyoom is just polling"? > systemd-oomd periodically polls PSI statistics for the system and those cgroups to decide when to take action. It's unclear if the docs for systemd-oomd are incorrect or misleading; I do see from the kernel.org link that the recommended usage pattern is to use the `poll` system call, which in this context would mean "not polling", if I understand correctly.
  - 100721 1y ago
    Unrelated to the topic, it seems awfully unintuitive to name a function ‘poll’ if the result is ‘not polling.’ I’m guessing there’s some history and maybe backwards-compatible rewrites?
    - unilynx 1y ago
      Poll takes a timeout parameter. ‘Not polling’ is just a really long timeout
    - friendzis 1y ago
      "Let the underlying platform do the polling and return once the condition is met"
    - CGamesPlay 1y ago
      Specifically, earlyoom’s README says it repeatedly checks (“periodically polls”) the memory pressure, using CPU each time even when there is no change. The “poll” system call waits for the kernel to notify the process that the file has changed, using no CPU until the call resolves. It’s unclear what systemd-oomd does, because it uses the phrase “periodically polls”,
nurettin 1y ago
Of course swap should be enabled. But oom killer has always allowed access to an otherwise unreachable system. The pause is there so you can impress your junior padawan who rushed to you in a hurry.
cmurf 1y ago
Some workloads may do better with zswap. Cache is compressed, and pages evicted to disk based swap on an LRU basis. The case of swap thrashing sounds like a misbehaving program, which can maybe be tamed by oomd. System responsiveness though needs a complete resource control regime in place, that preserves minimum resources for certain critical processes. This is done with cgroupsv2. By establishing minimum resources, the kernel will limit resources for other processes. Sure, they will suffer. That’s the idea.
RobRivera 1y ago
To learn tricks like this what resource do you recommend I read? System administrators handbook? (Still on my TOREAD queue)
- eitland 1y ago
  "The practice of System and Network administration" by Tom Limoncelli and Christine Hogan[1] was, together with "Principles of Network and Systems Administration" by Mark Burgess have probably been the books that influenced my approach to sysadmin the most. I still have them. Between them they covered at a high level (at least back when I was sysadmin before devops and Kubernets etc) anything and everything from - hardware, networks, monitoring, provisioning, server room locations in existing buildings, how to prepare server rooms - and so on up to hiring and firing sysadmins, salary negotiations[2], vendor negotiations and the first book even had a whole chapter dedicated to "Being happy" [1] There is a third author as well now, but those two were the ones that are on the cover of my book from 2005 and that I can remember [2] Has mostly worked well after I more or less left sysadmin behind as well
bouncycastle 1y ago
sometimes swap seems to accumulate even though there is plenty of ram. It is too "greedy" by default, probably set for desktops not servers in mind. Therefore it is better to always tune "vm.swappiness" to 1 in /etc/sysctl.conf You can also configure your web server / TCP stack buffers / file limits so they never allocate memory over the physical ram available. (eg. in nginx you can setup worker/connection limits and buffer sizes.)
icetank 1y ago
Yeah I had a few servers look up on me without any clear way to recovery because some app was eating up ram. I am ok with the server coming to a crawl as soon as the swap has to be used but at least it won't stop responding all together.
tarruda 1y ago
> Here is how I do it on my Hetzner bare-metal servers using Ansible: https://gist.github.com/fungiboletus/794a265cc186e79cd5eb2fe https://gist.github.com/fungiboletus/794a265cc186e79cd5eb2fe... It also works on VMs. As someone with zero ansible experience, can you elaborate on why a yaml list is better than a simple shell script with comments before each command?
1vuio0pswjnm7 1y ago
"You may want to enable earlyoom, so your whole server doesn't go down when a service goes bananas." Another option is to run BSD to avoid the Linux oom issue For example, I'm not using Hetzner but I run NetBSD entirely from memory (no disk, no swap) and it never "went down" when out of memory Looks like some people install FreeBSD and OpenBSD on Hetzner https://gist.github.com/c0m4r/142a0480de4258d5da94ce3a2380e8bb https://gist.github.com/c0m4r/142a0480de4258d5da94ce3a2380e8... https://computingforgeeks.com/how-to-install-freebsd-on-hetzner-root-server/ https://computingforgeeks.com/how-to-install-freebsd-on-hetz... https://web.archive.org/web/20231211052837if_/https://www.area536.com/posts/freebsd-on-hetzner/ https://web.archive.org/web/20231211052837if_/https://www.ar... https://community.hetzner.com/tutorials/freebsd-openzfs-via-linux-rescue https://community.hetzner.com/tutorials/freebsd-openzfs-via-... https://www.souji-thenria.net/posts/openbsd_hetzner/ https://www.souji-thenria.net/posts/openbsd_hetzner/ https://web.archive.org/web/20220814124443if_/https://blog.vx.sk/archives/353 https://web.archive.org/web/20220814124443if_/https://blog.v... https://www.blunix.com/blog/how-to-install-openbsd-on-hetzner-cloud-servers.html https://www.blunix.com/blog/how-to-install-openbsd-on-hetzne... https://gist.github.com/ctsrc/9a72bc9a0229496aab5e4d3745af0bb9 https://gist.github.com/ctsrc/9a72bc9a0229496aab5e4d3745af0b... If it is possible to boot Hetzner from a BSD install image using "Linux rescue mode"^1 then it should also possible to run NetBSD entirely from memory using custom kernel Every user is different but this is how I prefer to run UNIX-like OS for personal, recreational use; I find it more resilient 1. https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system/ https://docs.hetzner.com/robot/dedicated-server/troubleshoot... https://blog.tericcabrel.com/hetzner-rescue-mode-unlock-server/ https://blog.tericcabrel.com/hetzner-rescue-mode-unlock-serv... https://github.com/td512/rescue https://github.com/td512/rescue https://gainanov.pro/eng-blog/linux/hetzner-rescue-mode/ https://gainanov.pro/eng-blog/linux/hetzner-rescue-mode/ https://docs.hetzner.com/cloud/servers/getting-started/rescue-system/ https://docs.hetzner.com/cloud/servers/getting-started/rescu... ChromeOS has an interesting approach to Linux oom issues. Not sure it has ever been discussed on HN https://github.com/dct2012/chromeos-3.14/raw/chromeos-3.14/mm/low-mem-notify.c https://github.com/dct2012/chromeos-3.14/raw/chromeos-3.14/m...

cactusplant7374 1y ago

What's the performance hit from compressing ram?

speedgoose 1y ago
I haven’t scientifically measured, but you don’t compress the whole ram. It is more about reserving a part of the ram to have very fast swap. For an algorithm using the whole memory, that’s a terrible idea.
- sokoloff 1y ago
  > It is more about reserving a part of the ram to have very fast swap. I understand all of those words, but none of the meaning. Why would I reserve RAM in order to put fast swap on it?
  - vlovich123 1y ago
    Swap to disk involves a relatively small pipe (usually 10x smaller than RAM). So instead of paying the cost to page out to disk immediately, you create compressed pages and store that in a dedicated RAM region for compressed swap. This has a number of benefits: in practice more “active” space is freed up as unused pages are compressed and often compressible. Often times that can be freed application memory that is reserved within application space but in the free space of the allocator, especially if that allocator zeroes it those pages in the background, but even active application memory (eg if you have a browser a lot of the memory is probably duplicated many times across processes). So for a usually invisible cost you free up more system RAM. Additionally, the overhead of the swap is typically not much more than a memcpy even compressed which means that you get dedup and if you compressed erroneously (data still needed) paging it back in is relatively cheap. It also plays really well with disk swap since the least frequently used pages of that compressed swap can be flushed to disk leaving more space in the compressed RAM region for additional pages. And since you’re flushing retrieving compressed pages from disk you’re reducing writes on an SSD (longevity) and reducing read/write volume (less overhead than naiive direct swap to disk). Basically if you think of it as tiered memory, you’ve got registers, l1 cache, l2 cache, l3 cache, normal RAM, compressed swap RAM, disk swap - it’s an extra interim tier that makes the system more efficient.
- LargoLasskhyfv 1y ago
  >...but you don’t compress the whole ram. I do: https://postimg.cc/G8Gcp3zb https://postimg.cc/G8Gcp3zb (casualmeasurement.png)
YouAreWRONGtoo 1y ago
It's sometimes not a hit, because CPUs have caches and memory bandwidth is the limiting factor.
waynesonfire 1y ago
> zram, formerly called compcache, is a Linux kernel module for creating a compressed block device in RAM, i.e. a RAM disk with on-the-fly disk compression. The block device created with zram can then be used for swap or as a general-purpose RAM disk To clarify OP's represention of the tool, it compresses swap space not resident ram. Outside of niche use-cases, compressing swap has overall little utility.
- coppsilgold 1y ago
  Incorrect, with zram you swap ram to compressed ram. It has the benefit of absorbing memory leaks (which for whatever reason compress really well) and compressing stale memory pages. Under actual memory pressure performance will degrade. But in many circumstances where your powerful CPU is not fully utilized you can 2x or even 3x your effective RAM (you can opt for zstd compression). zram also enables you to make the trade-off of picking a more powerful CPU for the express purpose of multiplying your RAM if the workload is compatible with the idea. PS: On laptops/workstations, zram will not interfere with an SSD swap partition if you need it for hibernation. Though it will almost never be used for anything else if you configure your zram to be 2x your system memory.
  - masklinn 1y ago
    > Incorrect, with zram you swap ram to compressed ram. That reads like what they said? You reserve part of the RAM as a swap device, and memory is swapped from resident RAM to the swap ramdisk, as long as there’s space on there. And AFAIK linux will not move pages between swap devices because it doesn’t understand them beyond priority. Zswap actually seems strictly better in many cases (especially interactive computers / dev machines) as it can more flexibly grow / shrink, and can move pages between the compressed RAM cache and the disk swap.
aidenn0 1y ago
Depends on the algorithm (and how much CPU is in use); if you have a spare CPU, the faster algorithms can more-or-less keep up with your memory bandwidth, making the overhead negligible. And of course the overhead is zero when you don't page-out to swap.

levkk 1y ago

Yeah, no way. As soon as you hit swap, _most_ apps are going to have a bad, bad time. This is well known, so much so that all EC2 instances in AWS disable it by default. Sure, they want to sell you more RAM, but it's also just true that swap doesn't work for today's expectations. Maybe back in the 90s, it was okay to wait 2-3 seconds for a button click, but today we just assume the thing is dead and reboot.

gchamonlive 1y ago
How programs use ram also changed from the 90s. Back then they were written targeting machines that they knew would have a hard time fitting all their data in memory, so hitting swap wouldn't hurt perceived performance too drastically since many operations were already optimized to balance data load between memory and disk. Nowadays when a program hits swap it's not going to fallback to a different memory usage profile that prioritises disk access. It's going to use swap as if it were actual ram, so you get to see the program choking the entire system.
- winrid 1y ago
  Exactly. Nowadays, most web services are run in a GC'ed runtime. That VM will walk pointers all over the place and reach into swap all the time.
  - cogman10 1y ago
    Depends entirely on the runtime. If your GC is a moving collector, then absolutely this is something to watch out for. There are, however, a number of runtimes that will leave memory in place. They are effectively just calling `malloc` for the objects and `free` when the GC algorithm detects an object is dead. Go, the CLR, Ruby, Python, Swift, and I think node(?) all fit in this category. The JVM has a moving collector.
    - zozbot234 1y ago
      Every garbage collector has to constantly sift through the entire reference graph of the running program to figure out what objects have become garbage. Generational GC's can trace through the oldest generations less often, but that's about it. Tracing garbage collectors solve a single problem really really well - managing a complex, possibly cyclical reference graph, which is in fact inherent to some problems where GC is thus irreplaceable - and are just about terrible wrt. any other system-level or performance-related factor of evaluation.

shrubble 1y ago

It's always a good idea to have a tiny amount of swap just in case. Like 1GB.

akerl_ 1y ago
Why?
- angch 1y ago
  Like a highway brake failure ramp, you have room for handling failures gentler. So services don't just get outright killed. If you monitor your swap usage, any usage of swap gives you early warning that your services require more memory already. Gives you some time to upgrade, or tune services before it goes ka-boom.
  - akerl_ 1y ago
    If your memory usage is creeping up, the way you'll find out that you need more memory is by monitoring memory usage via the same mechanisms you'd hypothetically use to monitor your swap usage. If your memory usage spikes suddenly, a nominal amount of swap isn't stopping anything from getting killed; you're at best buying yourself a few seconds, so unless you spend your time just staring at the server, it'll be dead anyways.
- CGamesPlay 1y ago
  Because some portion of the RAM used by your daemons isn't actually being accessed, and using that RAM to store file cache is actually a better use than storing idle memory. The old rule about "as much swap as main memory" definitely doesn't hold any more, but a few GB to store unneeded wired memory to dedicate more room to file cache is still useful. As a small example from a default Ubuntu installation, "unattended-upgrades" is holding 22MB of RSS, and will not impact system performance at all if it spends next week swapped out. Bigger examples can be found in monolithic services where you don't use some of the features but still have to wire them into RAM. You can page those inactive sections of the individual process into swap, and never notice.

awesome_dude 1y ago

How do you get swap on a VPS?

justsomehnguy 1y ago
Search "linux enable swap in a file" To enable a swap file in Linux, first create the swap file using a command like sudo dd if=/dev/zero of=/swapfile bs=1G count=1 for a 1GB file. Then, set it up with sudo mkswap /swapfile and activate it using sudo swapon /swapfile. To make it permanent, add /swapfile swap swap defaults 0 0 to your /etc/fstab file.
- awesome_dude 1y ago
  Strongly suggest you try doing that on a VPS, then report back
  - ahepp 1y ago
    What do you think is going to happen? I tested it out on an ec2 instance just now and it seems to have worked as one would expect.
    - awesome_dude 1y ago
      EC2 != VPS
      - cmpxchg8b 1y ago
        They both offer virtualized guests under a hypervisor host. EC2 does have more offload specialization hardware but for the most part they are functionally equivalent, unless I'm missing something...
      - ahepp 1y ago
        What VPS do you think this would cause problems on? Why?
    - 1y ago

Bender 1y ago

Another option would be to have more memory that required over-engineer and to adjust the oom score per app, adding early kill weight to non critical apps and negative weight to important apps. oom_score_adj is already set to -1000 by OpenSSH for example. NSDJUST=$(pgrep -x nsd); echo -en '-378' > /proc/"${NSDJUST}"/oom_score_adj Another useful thing to do is effecively disable over-commit on all staging and production servers (0 ratio instead of 2 memory to fully disable as these do different things, memory 0 still uses formula) vm.overcommit_memory = 0 vm.overcommit_ratio = 0 Also use a formula to set min_free and reserved memory using a formula from Redhat that I do not have handy based on installed memory. min_free can vary from 512KB to 16GB depending on installed memory. vm.admin_reserve_kbytes = 262144 vm.user_reserve_kbytes = 262144 vm.min_free_kbytes = 1024000 At least that worked for me in about 50,000 physical servers for over a decade that were not permitted to have swap and installed memory varied from 144GB to 4TB of RAM. OOM would only occur when the people configuring and pushing code would massively over-commit and not account for memory required by the kernel. Not following best practices defined by Java and thats a much longer story. Another option is to limit memory per application in cgroups but that requires more explaining than I am putting in an HN comment. Another useful thing is to never OOM kill in the first place on servers that are only doing things in memory and need not commit anything to disk. So don't do this on a disked database. This is for ephemeral nodes that should self heal. Wait 60 seconds so drac/ilo can capture crash message and then earth shattering kaboom... # cattle vs kittens, mooooo... kernel.panic = 60 vm.panic_on_oom = 2 For a funny side note, those options can also be used as a holy hand grenade to intentionally unsafely reboot NFS diskless farms when failing over to entirely different NFS server clusters. setting panic to 15 mins, triggering OOM panic by setting min_free to 16TB at the command line via Ansible not in sysctl.conf, swapping clusters, arp storm and reconverge.

liqilin1567 1y ago
Thanks for sharing I think these are very useful suggestions.
benterix 1y ago
The lengths people will go to avoid k8s... (very easy on Hetzner Cloud BTW).

statictype 1y ago

Thanks for this. We resorted to setting ram thresholds in systemd. Is earlyoom a better solution than that to prevent an erratic process from making an instance unresposnsive?

dboreham 1y ago

Haven't used swap since 2010.

TheDong 1y ago

Even better than earlyoom is systemd-oomd[0] or oomd[1]. systemd-oomd and oomd use the kernel's PSI[2] information which makes them more efficient and responsive, while earlyoom is just polling. earlyoom keeps getting suggested, even though we have PSI now, just because people are used to using it and recommending it from back before the kernel had cgroups v2. [0]: https://www.freedesktop.org/software/systemd/man/latest/systemd-oomd.service.html https://www.freedesktop.org/software/systemd/man/latest/syst... [1]: https://github.com/facebookincubator/oomd https://github.com/facebookincubator/oomd [2]: https://docs.kernel.org/accounting/psi.html https://docs.kernel.org/accounting/psi.html

CGamesPlay 1y ago
"earlyoom is just polling"? > systemd-oomd periodically polls PSI statistics for the system and those cgroups to decide when to take action. It's unclear if the docs for systemd-oomd are incorrect or misleading; I do see from the kernel.org link that the recommended usage pattern is to use the `poll` system call, which in this context would mean "not polling", if I understand correctly.
- 100721 1y ago
  Unrelated to the topic, it seems awfully unintuitive to name a function ‘poll’ if the result is ‘not polling.’ I’m guessing there’s some history and maybe backwards-compatible rewrites?
  - unilynx 1y ago
    Poll takes a timeout parameter. ‘Not polling’ is just a really long timeout
  - friendzis 1y ago
    "Let the underlying platform do the polling and return once the condition is met"
  - CGamesPlay 1y ago
    Specifically, earlyoom’s README says it repeatedly checks (“periodically polls”) the memory pressure, using CPU each time even when there is no change. The “poll” system call waits for the kernel to notify the process that the file has changed, using no CPU until the call resolves. It’s unclear what systemd-oomd does, because it uses the phrase “periodically polls”,

nurettin 1y ago

Of course swap should be enabled. But oom killer has always allowed access to an otherwise unreachable system. The pause is there so you can impress your junior padawan who rushed to you in a hurry.

cmurf 1y ago

Some workloads may do better with zswap. Cache is compressed, and pages evicted to disk based swap on an LRU basis. The case of swap thrashing sounds like a misbehaving program, which can maybe be tamed by oomd. System responsiveness though needs a complete resource control regime in place, that preserves minimum resources for certain critical processes. This is done with cgroupsv2. By establishing minimum resources, the kernel will limit resources for other processes. Sure, they will suffer. That’s the idea.

RobRivera 1y ago

To learn tricks like this what resource do you recommend I read? System administrators handbook? (Still on my TOREAD queue)

eitland 1y ago
"The practice of System and Network administration" by Tom Limoncelli and Christine Hogan[1] was, together with "Principles of Network and Systems Administration" by Mark Burgess have probably been the books that influenced my approach to sysadmin the most. I still have them. Between them they covered at a high level (at least back when I was sysadmin before devops and Kubernets etc) anything and everything from - hardware, networks, monitoring, provisioning, server room locations in existing buildings, how to prepare server rooms - and so on up to hiring and firing sysadmins, salary negotiations[2], vendor negotiations and the first book even had a whole chapter dedicated to "Being happy" [1] There is a third author as well now, but those two were the ones that are on the cover of my book from 2005 and that I can remember [2] Has mostly worked well after I more or less left sysadmin behind as well

bouncycastle 1y ago

sometimes swap seems to accumulate even though there is plenty of ram. It is too "greedy" by default, probably set for desktops not servers in mind. Therefore it is better to always tune "vm.swappiness" to 1 in /etc/sysctl.conf You can also configure your web server / TCP stack buffers / file limits so they never allocate memory over the physical ram available. (eg. in nginx you can setup worker/connection limits and buffer sizes.)

icetank 1y ago

Yeah I had a few servers look up on me without any clear way to recovery because some app was eating up ram. I am ok with the server coming to a crawl as soon as the swap has to be used but at least it won't stop responding all together.

tarruda 1y ago

> Here is how I do it on my Hetzner bare-metal servers using Ansible: https://gist.github.com/fungiboletus/794a265cc186e79cd5eb2fe https://gist.github.com/fungiboletus/794a265cc186e79cd5eb2fe... It also works on VMs. As someone with zero ansible experience, can you elaborate on why a yaml list is better than a simple shell script with comments before each command?

1vuio0pswjnm7 1y ago

"You may want to enable earlyoom, so your whole server doesn't go down when a service goes bananas." Another option is to run BSD to avoid the Linux oom issue For example, I'm not using Hetzner but I run NetBSD entirely from memory (no disk, no swap) and it never "went down" when out of memory Looks like some people install FreeBSD and OpenBSD on Hetzner https://gist.github.com/c0m4r/142a0480de4258d5da94ce3a2380e8bb https://gist.github.com/c0m4r/142a0480de4258d5da94ce3a2380e8... https://computingforgeeks.com/how-to-install-freebsd-on-hetzner-root-server/ https://computingforgeeks.com/how-to-install-freebsd-on-hetz... https://web.archive.org/web/20231211052837if_/https://www.area536.com/posts/freebsd-on-hetzner/ https://web.archive.org/web/20231211052837if_/https://www.ar... https://community.hetzner.com/tutorials/freebsd-openzfs-via-linux-rescue https://community.hetzner.com/tutorials/freebsd-openzfs-via-... https://www.souji-thenria.net/posts/openbsd_hetzner/ https://www.souji-thenria.net/posts/openbsd_hetzner/ https://web.archive.org/web/20220814124443if_/https://blog.vx.sk/archives/353 https://web.archive.org/web/20220814124443if_/https://blog.v... https://www.blunix.com/blog/how-to-install-openbsd-on-hetzner-cloud-servers.html https://www.blunix.com/blog/how-to-install-openbsd-on-hetzne... https://gist.github.com/ctsrc/9a72bc9a0229496aab5e4d3745af0bb9 https://gist.github.com/ctsrc/9a72bc9a0229496aab5e4d3745af0b... If it is possible to boot Hetzner from a BSD install image using "Linux rescue mode"^1 then it should also possible to run NetBSD entirely from memory using custom kernel Every user is different but this is how I prefer to run UNIX-like OS for personal, recreational use; I find it more resilient 1. https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system/ https://docs.hetzner.com/robot/dedicated-server/troubleshoot... https://blog.tericcabrel.com/hetzner-rescue-mode-unlock-server/ https://blog.tericcabrel.com/hetzner-rescue-mode-unlock-serv... https://github.com/td512/rescue https://github.com/td512/rescue https://gainanov.pro/eng-blog/linux/hetzner-rescue-mode/ https://gainanov.pro/eng-blog/linux/hetzner-rescue-mode/ https://docs.hetzner.com/cloud/servers/getting-started/rescue-system/ https://docs.hetzner.com/cloud/servers/getting-started/rescu... ChromeOS has an interesting approach to Linux oom issues. Not sure it has ever been discussed on HN https://github.com/dct2012/chromeos-3.14/raw/chromeos-3.14/mm/low-mem-notify.c https://github.com/dct2012/chromeos-3.14/raw/chromeos-3.14/m...