7 ms·
My personal rules of thumb for Linux systems. YMMV. * If you need a low-latency server or workstation and all of your processes are killable (i.e. they can be
by mwpmaybe 10y ago
My personal rules of thumb for Linux systems. YMMV.
* If you need a low-latency server or workstation and all of your processes are killable (i.e. they can be easily/automatically restarted without data loss): disable swap.
* If you need a low-latency server or workstation and some of your processes are not killable (e.g. databases): enable swap and set vm.swappiness to 0.
* SSD-backed desktops and other servers and workstations: enable swap and set vm.swappiness to 1 (for NAND flash longevity).
* Disk-backed desktops and other servers and workstations: accept the system/distro defaults, typically swap enabled with vm.swappiness set to 60. You can and likely should lower vm.swappiness to 10 or so if you have a ton of RAM relative to your workload.
* If your server or workstation has a mix of killable and non-killable processes, use oom_score_adj to protect the non-killable processes.
* Monitor systems for swap (page-out) activity.
- gregmac 10y agoFor the curious (I was): * vm.swappiness = 0 The kernel will swap only to avoid an out of memory condition, when free memory will be below vm.min_free_kbytes limit. * vm.swappiness = 1 Minimum amount of swapping without disabling it entirely. * vm.swappiness = 60 The default value. * vm.swappiness = 100 The kernel will swap aggressively. https://en.wikipedia.org/wiki/Swappiness https://en.wikipedia.org/wiki/Swappiness
- Jedd 10y ago> vm.swappiness = 0 The kernel will swap only to avoid an out of memory condition, when free memory will be below vm.min_free_kbytes limit. This is not the case. It used to be the case, but changed in kernel version 3.5-rc1 (2012 ish) There was a discussion about this on HN a few weeks ago: https://news.ycombinator.com/item?id=13511086 https://news.ycombinator.com/item?id=13511086 And there's a blog post on the percona website about how this rather bizarre change bit them: https://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel/ https://www.percona.com/blog/2014/04/28/oom-relation-vm-swap... I call it bizarre because (as I wrote in that other HN thread) a) it changed the behaviour of lots of production systems in a surprising way, and b) if you want to ensure your processes never swap you already had the option to not have a swap file or partition.
- sbuttgereit 10y agoWasn't there a Debian/Ubuntu thing recently where vm.swappiness = 0 had a behavior change which increased the number of incidents of the OOM killer stomping on things like database processes? (Maybe it wasn't so new... https://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel/ https://www.percona.com/blog/2014/04/28/oom-relation-vm-swap...)
- mwpmaybe 10y agoThank you for sharing this. There's an interesting conversation thread in the comments on that post. It's a little over my head, but my takeaway is that with the kernel change, in an OOM event, MySQL is unable to be swapped out due to the type(s) of memory pages it's using, so the kernel is forced to kill it (or itself). In practice, it's relatively straightforward to tune MySQL/MariaDB for a certain memory allocation, and if it's on a shared host, oom_score_adj can be set to protect it.
- feld 10y agoCan you not protect processes from the oomkiller? This is trivial and very useful on FreeBSD https://www.freebsd.org/cgi/man.cgi?query=protect&sektion=1 https://www.freebsd.org/cgi/man.cgi?query=protect&sektion=1
- mwpmaybe 10y agoYes, with oom_score_adj[0], which I've mentioned several times. Setting it to -1000 for a process protects it from OOM killing. 0. http://man7.org/linux/man-pages/man5/proc.5.html http://man7.org/linux/man-pages/man5/proc.5.html
- feld 10y agothat looks painful to use. Do Linux distros let you automatically protect services? E.g. On FreeBSD: mysql_enable="YES" mysql_oomprotect="YES" Now every time you start the MySQL service it's automatically protected
- nisa 10y agoIf you are on the experimental side: There is also zram (just swap in memory lz4/lzo compressed) and zswap (compressed cache in memory for swap pages before hitting disk) that needs a real swap device but compresses pages beforehand. I run zswap on my Desktop and on a few servers and it gives you some more time before the oom killer comes and the system feels a bit longer responsive. zram is a nice idea but quite a beast in practice (at least on MIPS with 32mb RAM) sys constantly at 100% if you need it and other quirks. Maybe it got better or I did something wrong. But if you need an in-memory compressed block-device it's pretty great - you can just format it with ext4 and have a lz4 compressed tmpfs.
- mxvzr 10y agoI use zram rather than a regular swap partition on all my laptops (because I'd rather not swap on SSDs) and desktops (same reason and/or there is an absurd amount of RAM to begin with). I also hear that most chromebooks use zram too (you really don't want to be swapping on that eMMC memory). I set it up with one zram device per CPU core for a total space of ~20% available RAM. No performance issues w/ zram so far so I haven't felt the need to change the compression algorithm.
- nerdponx 10y agoFirst I've heard of either. How would I set these up?
- mxvzr 10y agoYou can setup zram like this. Typically you'll want to make a service for it since it needs to run on every boot. # modprobe zram num_devices=1 # echo 1G > /sys/block/zram0/disksize # mkswap zram0 /dev/zram0 -L zram0 # swapon -p 100 /dev/zram0 Official documentation here: https://www.kernel.org/doc/Documentation/blockdev/zram.txt https://www.kernel.org/doc/Documentation/blockdev/zram.txt
- isr 10y agozram has worked fine on my chromebooks. This is with running multiple chroots - and I have hit the oom killer a number of times (when even zram swap wasn't enough). Until you actually run out of memory, zram seems very much a set-and-forget type of thing. No babysitting required. tl;dr: it does what it says on the tin, and ... with minimal cpu impact.
- kalleboo 10y ago> * SSD-backed desktops and other servers and workstations: enable swap and set vm.swappiness to 1 (for NAND flash longevity). Is this that big of a worry? I have a 5-year old SSD in my daily driver laptop, on OS X which loooves to swap out anything it can to gain memory for disk cache, and I'm still barely 15% into the SSD wearout.