5 ms·

I'm very much against changing kernel settings in production servers without really understanding the implications. Take for example the "swappiness = 0", most

by lazyant 12y ago

I'm very much against changing kernel settings in production servers without really understanding the implications. Take for example the "swappiness = 0", most likely what you think it does it's not what it does.

voidlogic 12y ago
vm.swapiness is not magic: http://en.wikipedia.org/wiki/Swappiness http://en.wikipedia.org/wiki/Swappiness People should of course learn about these setting as they change them, but many of the suggested values really are improvements for lots of use cases.
- lazyant 12y ago
  swapiness=0 in some Linux versions disables swapping, also even with swapiness=0 and plenty of RAM, kernel sometimes will swap; people put too much faith in these values intead of doing proper capacity planning/monitoring imhho
mackal 12y ago
Yeah, my swappiness is set to default (60) I still never use my swap space :/
jcurbo 12y ago
Indeed, I wish this post had his rationale behind the changes and descriptions of what each change does.
phunge 12y ago
Looking through his list though, most of the settings are less controversial than swappiness. I do wish people were more rigorous about changing swappiness. Maybe if swappiness=0 was documented as "pageout every byte of /usr/sbin/sshd and /lib/libc.so before you pageout a single byte of java's bloated heap" then people would be less eager to apply it.
- sp332 12y ago
  Since kernel 3.5-rc1, vm.swappiness=0 will disable swapping entirely. https://news.ycombinator.com/item?id=7660385 https://news.ycombinator.com/item?id=7660385
- bcoates 12y ago
  Are there people running servers that actually swap without it being a disaster? For regular 'cattle not pets' servers I just decline to create a swap partition and vm.panic_on_oom = 1. Java's bloated heap, for example, has easy to use startup options that cause the Java process to use a deterministic amount of RAM, as does any other sane program that uses nontrivial amounts of memory. OOMing (or swapping) a server almost always indicates an error that needs human intervention, so cleanly killing/rebooting the faulting system and raising the alarm in your monitoring system is the right thing to do.
  - AlTobey 12y ago
    Amen.
  - scott_s 12y ago
    "Swapping" does not imply that the system is only swapping. Yes, if the system is only swapping - or, more accurately, if kswapd is what's pegging your CPUs at 100% - then you have problems. But the kernel will try to swap out unused pages to make room for the disk cache. If you turn this off, it won't do that, and you could end up hitting the disk more. Modern systems are usually teetering near the edge of using all physical RAM. This is by design; unused RAM is wasted. You want to use as much RAM as possible to avoid going to disk. What you call "OOMing" is when the applications on the system require more physical RAM than what exists. This is independent from the swappiness ratio.
  - phunge 12y ago
    Yes I believe swap makes some workloads faster. The goal of virtual memory is to apportion physical memory to the things that need it most -- keep frequently used data in RAM, and pageout things which aren't frequently used. This makes things go faster. When you disable swap, you constrain VM's ability to do this -- now it must keep ALL anonymous memory in RAM, and it will pageout file-backed memory instead. Even if that file-backed memory is much hotter. I came to this conclusion by following the kernel community, starting with the "why swap at all" flamewar on LKML. See this response from Nick Piggin http://marc.info/?t=108555368800003&r=1&w=2 http://marc.info/?t=108555368800003&r=1&w=2 who is a fairly prominent kernel developer. Nothing I've read from the horses mouth has refuted this since then. This is true even on systems with gobs of memory. You're worried about systems which grind to a halt under memory pressure, which is unquestionably a concern. The thing is, disabling swap doesn't fix this. As soon as you're paging out important file-backed pages (like libc), your system is going to grind to a halt anyway. And disabling swap can't prevent this -- (same point from a VM developer here http://marc.info/?l=linux-kernel&m=108557438107853&w=2 http://marc.info/?l=linux-kernel&m=108557438107853&w=2). To really give your prod systems a safety net, you need to (a) lock important memory (like, say, SSH and libc) in memory or (b) constrain processes which hog memory with (e.g.) memory cgroups. IMHO cgroups/containers are a better solution.
guidedlight 12y ago
Isn't "swappiness = 0" recommended for SSD-based swap partitions to reduce SSD wear?!?
- emeraldd 12y ago
  I've seen swappiness = 1 as a recommendation rather than 0.
- khc 12y ago
  if you want to reduce SSD wear, just don't have a swap partition on SSD (or by not having a swap partition at all if you only have SSD)
- Rapzid 12y ago
  It's like a new pair of shoes; give it a few weeks and you won't mind the wear so much.
- simoncion 12y ago
  Relatively modern SSDs will write at least 500TB before dying: http://techreport.com/review/26523/the-ssd-endurance-experiment-casualties-on-the-way-to-a-petabyte http://techreport.com/review/26523/the-ssd-endurance-experim... I've an SSD that's been running in my development Linux laptop for very, very close to three years. The drive houses a couple of encrypted swap partitions along with the rest of the system. According to the SMART attributes, I've written 18TB to it in that time. Don't worry about SSD wear. Really, don't. Either you'll get a drive that succumbs to crib death or super-shitty v1.0 firmware, or you'll get a drive that will last until long after you outgrow it.
AlTobey 12y ago
My recommendation for any server with enough RAM is to disable swap. This works on just about every system whether swap is configured or not. That said, on my Linode I have swappiness=1 and a few GB of swap on the SSD since it is memory-constrained. In any case, I've been running vm.swappiness=0 or 1 on nearly every machine I touch for many years now and have yet to see any problems. Swap is almost always a bad idea on modern machines. Thanks for reading!
- aidenn0 12y ago
  I strongly disagree, and prefer the swappiness at the default level with a modestly sized swap file (1-2GB). There are lots of poorly behaved applications that will allocate RAM that they never touch again. If you turn down swappiness you end up wasting that RAM which could be better used for disk cache. I currently have 1.2GB of swap in use on a machine that is not doing any active swapping at all; that's 1.2GB more space for caching. [edit for spelling]
  - scott_s 12y ago
    Agreed. I'm not sure if the author of the original article really understands how modern operating systems deal with memory. The point of swapping pages to disk is to avoid hitting the disk in the long run. See: http://archive.today/FKlQ http://archive.today/FKlQ
    - edwintorok 12y ago
      Except thats not what causes problems in practice. You get problems with swap when your working set size exceeds your RAM. You'll always be hitting the disk as long as the offending application keeps using too much memory, and it slows down the entire system not just that one application. When you try to SSH in/log in that requires reading files from disk (or swapping things back in) which can take quite a while due the backlog caused by swapping. If you ran out of memory and malloc returned NULL that would be better, however a lot of applications rely on overcommit so there is no good answer: * you can run with RAM + as much swap to make all applications happy and overcommit off, and accept the long delays caused by swapping * run with just RAM, overcommit on and no swap, and accept that your applications may be killed by the OOM killer anytime

by lazyant 12y ago

voidlogic 12y ago

vm.swapiness is not magic: http://en.wikipedia.org/wiki/Swappiness http://en.wikipedia.org/wiki/Swappiness People should of course learn about these setting as they change them, but many of the suggested values really are improvements for lots of use cases.

lazyant 12y ago
swapiness=0 in some Linux versions disables swapping, also even with swapiness=0 and plenty of RAM, kernel sometimes will swap; people put too much faith in these values intead of doing proper capacity planning/monitoring imhho

mackal 12y ago

Yeah, my swappiness is set to default (60) I still never use my swap space :/

jcurbo 12y ago

Indeed, I wish this post had his rationale behind the changes and descriptions of what each change does.

phunge 12y ago

Looking through his list though, most of the settings are less controversial than swappiness. I do wish people were more rigorous about changing swappiness. Maybe if swappiness=0 was documented as "pageout every byte of /usr/sbin/sshd and /lib/libc.so before you pageout a single byte of java's bloated heap" then people would be less eager to apply it.

sp332 12y ago
Since kernel 3.5-rc1, vm.swappiness=0 will disable swapping entirely. https://news.ycombinator.com/item?id=7660385 https://news.ycombinator.com/item?id=7660385
bcoates 12y ago
Are there people running servers that actually swap without it being a disaster? For regular 'cattle not pets' servers I just decline to create a swap partition and vm.panic_on_oom = 1. Java's bloated heap, for example, has easy to use startup options that cause the Java process to use a deterministic amount of RAM, as does any other sane program that uses nontrivial amounts of memory. OOMing (or swapping) a server almost always indicates an error that needs human intervention, so cleanly killing/rebooting the faulting system and raising the alarm in your monitoring system is the right thing to do.
- AlTobey 12y ago
  Amen.
- scott_s 12y ago
  "Swapping" does not imply that the system is only swapping. Yes, if the system is only swapping - or, more accurately, if kswapd is what's pegging your CPUs at 100% - then you have problems. But the kernel will try to swap out unused pages to make room for the disk cache. If you turn this off, it won't do that, and you could end up hitting the disk more. Modern systems are usually teetering near the edge of using all physical RAM. This is by design; unused RAM is wasted. You want to use as much RAM as possible to avoid going to disk. What you call "OOMing" is when the applications on the system require more physical RAM than what exists. This is independent from the swappiness ratio.
- phunge 12y ago
  Yes I believe swap makes some workloads faster. The goal of virtual memory is to apportion physical memory to the things that need it most -- keep frequently used data in RAM, and pageout things which aren't frequently used. This makes things go faster. When you disable swap, you constrain VM's ability to do this -- now it must keep ALL anonymous memory in RAM, and it will pageout file-backed memory instead. Even if that file-backed memory is much hotter. I came to this conclusion by following the kernel community, starting with the "why swap at all" flamewar on LKML. See this response from Nick Piggin http://marc.info/?t=108555368800003&r=1&w=2 http://marc.info/?t=108555368800003&r=1&w=2 who is a fairly prominent kernel developer. Nothing I've read from the horses mouth has refuted this since then. This is true even on systems with gobs of memory. You're worried about systems which grind to a halt under memory pressure, which is unquestionably a concern. The thing is, disabling swap doesn't fix this. As soon as you're paging out important file-backed pages (like libc), your system is going to grind to a halt anyway. And disabling swap can't prevent this -- (same point from a VM developer here http://marc.info/?l=linux-kernel&m=108557438107853&w=2 http://marc.info/?l=linux-kernel&m=108557438107853&w=2). To really give your prod systems a safety net, you need to (a) lock important memory (like, say, SSH and libc) in memory or (b) constrain processes which hog memory with (e.g.) memory cgroups. IMHO cgroups/containers are a better solution.

guidedlight 12y ago

Isn't "swappiness = 0" recommended for SSD-based swap partitions to reduce SSD wear?!?

emeraldd 12y ago
I've seen swappiness = 1 as a recommendation rather than 0.
khc 12y ago
if you want to reduce SSD wear, just don't have a swap partition on SSD (or by not having a swap partition at all if you only have SSD)
Rapzid 12y ago
It's like a new pair of shoes; give it a few weeks and you won't mind the wear so much.
simoncion 12y ago
Relatively modern SSDs will write at least 500TB before dying: http://techreport.com/review/26523/the-ssd-endurance-experiment-casualties-on-the-way-to-a-petabyte http://techreport.com/review/26523/the-ssd-endurance-experim... I've an SSD that's been running in my development Linux laptop for very, very close to three years. The drive houses a couple of encrypted swap partitions along with the rest of the system. According to the SMART attributes, I've written 18TB to it in that time. Don't worry about SSD wear. Really, don't. Either you'll get a drive that succumbs to crib death or super-shitty v1.0 firmware, or you'll get a drive that will last until long after you outgrow it.

AlTobey 12y ago

My recommendation for any server with enough RAM is to disable swap. This works on just about every system whether swap is configured or not. That said, on my Linode I have swappiness=1 and a few GB of swap on the SSD since it is memory-constrained. In any case, I've been running vm.swappiness=0 or 1 on nearly every machine I touch for many years now and have yet to see any problems. Swap is almost always a bad idea on modern machines. Thanks for reading!

aidenn0 12y ago
I strongly disagree, and prefer the swappiness at the default level with a modestly sized swap file (1-2GB). There are lots of poorly behaved applications that will allocate RAM that they never touch again. If you turn down swappiness you end up wasting that RAM which could be better used for disk cache. I currently have 1.2GB of swap in use on a machine that is not doing any active swapping at all; that's 1.2GB more space for caching. [edit for spelling]
- scott_s 12y ago
  Agreed. I'm not sure if the author of the original article really understands how modern operating systems deal with memory. The point of swapping pages to disk is to avoid hitting the disk in the long run. See: http://archive.today/FKlQ http://archive.today/FKlQ
  - edwintorok 12y ago
    Except thats not what causes problems in practice. You get problems with swap when your working set size exceeds your RAM. You'll always be hitting the disk as long as the offending application keeps using too much memory, and it slows down the entire system not just that one application. When you try to SSH in/log in that requires reading files from disk (or swapping things back in) which can take quite a while due the backlog caused by swapping. If you ran out of memory and malloc returned NULL that would be better, however a lot of applications rely on overcommit so there is no good answer: * you can run with RAM + as much swap to make all applications happy and overcommit off, and accept the long delays caused by swapping * run with just RAM, overcommit on and no swap, and accept that your applications may be killed by the OOM killer anytime