4 ms·

Depends? If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and beco

by theamk 5y ago

Depends? If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on. If you have a latency-critical server, don't use swap. With swap, some data will get swapped out, and you will have sudden latency trying to access it. Without swap, it may still swap out the code, but the code is usually much smaller than the data, plus it is all pretty hot. If occasionally you have to process a very large datasets, and you are willing to wait a minute or twenty, you may enable swap. But be patient, and make sure your ssh has no timeout which can be hit. If you have a small amount of RAM, enable swap. It would be slow, but at least you would be able to do something without crashing.

Teletio 5y ago
There are CICD builds out there which consume much more resources and time were just killing one part of the build would destroy the work of hours. Not sure why you wouldn't want swap for it? It will allow you to fine-tune the build later and give that build a realistic chance to finish
- theamk 5y ago
  Because once swap activates, build now takes hours instead of tens of minutes. So it would timeout anyway, but only after wasting lots of resources. And even if you increase the timeout a lot instead, your machine how has a bunch of things swapped out, so now your tests timeout, which is even worse. Yes, killing that part of the build did destroy the work of hours. It was still better to disable the swap than try to "ride it out".
  - Teletio 5y ago
    Things don't just take hours longer just because the Linux kernel throws out a few pages which haven't been used for a while. And it also totally depends on how much memory is missing. I still prefer to have something taking 20 minutes longer instead of failing and fine-tuning the resources after it.
AlisdairO 5y ago
> If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on. This doesn't really work that well. It's true that if you enable swap and have significant memory pressure for any extended period your machine will grind to a halt, but this will _also_ happen if you don't use swap and rely on the Linux OOM killer. Indeed, despite the lack of swap, as part of trying to avoid OOM killing applications, Linux will grind the hell out of your disk - because it will drop executable pages out of RAM to free up space, then read them back in again on demand. As memory pressure increases, the period of time between dropping the page and reading it back in again becomes very short, and all your applications run super slowly. An easy solution to this is a userspace OOM-kill daemon like https://facebookmicrosites.github.io/oomd/ https://facebookmicrosites.github.io/oomd/ . This works on pressure stall information, so it knows when your system is genuinely struggling to free up memory. On the historical fleets I've worked on pre-OOMD/PSI, a reasonable solution was to enable swap (along with appropriate cgroups), but target only allowing brief periods of swapin/out. This gives you two advantages: * allows you to ride out brief periods of memory overconsumption * allows genuinely rarely accessed memory to be swapped out, giving you more working space compared to having no swap
- lazide 5y ago
  Eh, I’ve never seen a machine actually use any notable amount of swap and not be functionally death spiraling. I’m sure someone somewhere is able to use swap and not have the machine death spiral, but from desktop to servers? It’s never been me. I always disable swap for this reason, and it’s always been the better choice. Not killing something off when you get to that point ASAP is a losing bargain.
  - taeric 5y ago
    Don't mistake every machine you have seen death spiraling using swap, with every machine using swap as death spiraling. Notably, how many machines did you not have to look at, because the swap was doing just fine?

by theamk 5y ago

Teletio 5y ago

There are CICD builds out there which consume much more resources and time were just killing one part of the build would destroy the work of hours. Not sure why you wouldn't want swap for it? It will allow you to fine-tune the build later and give that build a realistic chance to finish

theamk 5y ago
Because once swap activates, build now takes hours instead of tens of minutes. So it would timeout anyway, but only after wasting lots of resources. And even if you increase the timeout a lot instead, your machine how has a bunch of things swapped out, so now your tests timeout, which is even worse. Yes, killing that part of the build did destroy the work of hours. It was still better to disable the swap than try to "ride it out".
- Teletio 5y ago
  Things don't just take hours longer just because the Linux kernel throws out a few pages which haven't been used for a while. And it also totally depends on how much memory is missing. I still prefer to have something taking 20 minutes longer instead of failing and fine-tuning the resources after it.

AlisdairO 5y ago

> If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on. This doesn't really work that well. It's true that if you enable swap and have significant memory pressure for any extended period your machine will grind to a halt, but this will _also_ happen if you don't use swap and rely on the Linux OOM killer. Indeed, despite the lack of swap, as part of trying to avoid OOM killing applications, Linux will grind the hell out of your disk - because it will drop executable pages out of RAM to free up space, then read them back in again on demand. As memory pressure increases, the period of time between dropping the page and reading it back in again becomes very short, and all your applications run super slowly. An easy solution to this is a userspace OOM-kill daemon like https://facebookmicrosites.github.io/oomd/ https://facebookmicrosites.github.io/oomd/ . This works on pressure stall information, so it knows when your system is genuinely struggling to free up memory. On the historical fleets I've worked on pre-OOMD/PSI, a reasonable solution was to enable swap (along with appropriate cgroups), but target only allowing brief periods of swapin/out. This gives you two advantages: * allows you to ride out brief periods of memory overconsumption * allows genuinely rarely accessed memory to be swapped out, giving you more working space compared to having no swap

lazide 5y ago
Eh, I’ve never seen a machine actually use any notable amount of swap and not be functionally death spiraling. I’m sure someone somewhere is able to use swap and not have the machine death spiral, but from desktop to servers? It’s never been me. I always disable swap for this reason, and it’s always been the better choice. Not killing something off when you get to that point ASAP is a losing bargain.
- taeric 5y ago
  Don't mistake every machine you have seen death spiraling using swap, with every machine using swap as death spiraling. Notably, how many machines did you not have to look at, because the swap was doing just fine?