5 ms·

Generally yes. Swapping changes the perf characteristics of that process (and often any other process on the same machine) in unpredictable ways. It's better to

Generally yes. Swapping changes the perf characteristics of that process (and often any other process on the same machine) in unpredictable ways. It's better to have predictable process termination -- with instrumentation describing what went wrong, so capacity planning and resource quotas can be updated. The process failure would generally be compensated-for at a higher level, anyway.

Galanwe 7y ago
> “process failure would generally be compensated-for at a higher level” I don’t quite get it. What does it mean? You re-run the process at an other time? On an other machine? I personally do have swap on all my production machines. I’m not relying on it, but it acts as a safety net in case unexpected memory usage sneak in some jobs. This goes with a sane periodic review of host metrics of course to ensure safety nets do not become the norm. I much prefer a slightly delayed job than a failed job.
- avianlyric 7y ago
  I guess it depends on the complexity of your distributed system (assuming you’re operating one). We prefer to have job OOM kill and get retried elsewhere (which could be a completely different machine) and we have plenty of infrastructure that makes this trivial. This infrastructure also deals with other types of partial failure, such as complete machine failure. As mentioned above, paging introduces strange pref behaviour. Which may not always be important, but if your working under tight latency requirements then paging can push you over that boundary. That sounds strange when we’re happy to see jobs die entirely (that screws latency). But the issue with paging is you have no idea when it’s gonna hit you, and may impact a job that’s behaving perfectly fine, except something got paged out by a badly behaving job. Ultimately disabling paging is a really good tool for limiting the blast radius of bad behaviour, and making cause-and-effect highly correlated (oh look thing X just OOMed, probably means thing X consumed too much memory. Rather than thing Y has strange tail latency because thing X keeps consuming too much memory). It’s failing fast, but for memory rather than exceptions.
  - Galanwe 7y ago
    I get your point, and I think we agree here. I mainly wanted to argue against the original statement, which was quite broad “high performance machines running in a production environments will absolutely have swap disabled”. Would you agree to rephrase both our arguments by: - paging incurs seemingly random performance degradation of processes and should be avoided - if you have a form of task queue/job distribution system which handles automatic re-run, and can afford at no business cost to restart a process from scratch, then disabling swapping allows fail fast behaviour - otherwise swapping can be used as a safety net for programs that would be better off slightly late than restarted from scratch - both scenarios require sane monitoring of process behaviours, to catch symptomatic failures/restart in case 1) and recurring swap usage in case 2)
  - AdamJacobMuller 7y ago
    > if your working under tight latency requirements then paging can push you over that boundary Even if you're not under tight requirements, swap can do strange things. I've actually seen situations where hitting swap, even trivially, can cause massive increases in latency. I'm talking about jobs which took 10s of milliseconds to complete now taking multiple 10s of seconds. I've even seem some absurdly bad memory management where Linux will make very very poor choices about what to page out. > Ultimately disabling paging is a really good tool for limiting the blast radius of bad behaviour 1000% agreed. Fail fast rather than fail slowly.
- rkagerer 7y ago
  >> “process failure would generally be compensated-for at a higher level” > I don’t quite get it. What does it mean? You re-run the process at an other time? On an other machine? I expect the commenter means whatever thing kicked off this job will detect it failed and do something about it (try it again elsewhere, log an alert triggering an admin to go provision more capacity, etc.) Their view is that a failure is easier to troubleshoot and fix than success with intermittently anomalous characteristics.
  - Galanwe 7y ago
    Got it. To be honest, what I really wanted here is to force the commenter on agreeing that whatever the architecture/framework/etc., there is no magical solution, and saying that it will "be compensated-for at a higher level" just hides the only 2 real possibilities: - Kill the process and restart it somewhere else (no swap) - Wait for the process to finish anyway (using swap) Both choices have their own pros and cons, as discussed in other comments. But arguments such as "it's better to have predictable process termination", and "under tight latency requirements then paging can push you over that boundary" are exposed as much less meaningful once the problem is stated in simple terms. If the process indeed uses more memory than planned, then you have no guarantee than restarting it will work this time. Worst case, you can even have to allocate urgently a new node to handle that 1MB of memory above what was available. Not sure that is really the best solution "under tight latency requirements". My experience would indeed be the reverse: if you are faced with unexpected memory usage growth, then having some swap on the side to allow your process to finish makes for a much smoother production system than having to restart it all from scratch somewhere else/at an other time. In my experience, production is not the place where you want to enforce ideological concepts such as "a process shall not use more memory than what was planned". Production is the place where you make compromises and account for everything that slipped through these concepts, because you have no choice: it _has_ to run. Staging would be a good place for this though, as it's much less critical if something fails, and you have more time to fix and account for it. I really don't want to debug/profile/release/deploy a program in a hurry just because somehow it ended up using more memory than planned. (or convince infrastructure to deploy a new node type ASAP to handle that new memory constraint). Now one could argue that such unplanned memory consumption should have been caught earlier in the development process, to which I would reply: - Not everyone is able to properly test programs in a production like environment, especially when dealing with resource-hungry processes. In our case for instance a typical staging machine would be a downscaled version of the production one (less cores, less memory, less bandwidth). - I tend to plan _also_ for what should _not_ happen (because it will happen). Even with a perfect environment to calibrate programs before a release, there will be releases in which the memory usage was not accounted for. Guaranteed.
- mimi89999 7y ago
  I have `vm.swappiness = 1` for that. Who would want a job to be OOM killed?

Galanwe 7y ago

> “process failure would generally be compensated-for at a higher level” I don’t quite get it. What does it mean? You re-run the process at an other time? On an other machine? I personally do have swap on all my production machines. I’m not relying on it, but it acts as a safety net in case unexpected memory usage sneak in some jobs. This goes with a sane periodic review of host metrics of course to ensure safety nets do not become the norm. I much prefer a slightly delayed job than a failed job.

avianlyric 7y ago
I guess it depends on the complexity of your distributed system (assuming you’re operating one). We prefer to have job OOM kill and get retried elsewhere (which could be a completely different machine) and we have plenty of infrastructure that makes this trivial. This infrastructure also deals with other types of partial failure, such as complete machine failure. As mentioned above, paging introduces strange pref behaviour. Which may not always be important, but if your working under tight latency requirements then paging can push you over that boundary. That sounds strange when we’re happy to see jobs die entirely (that screws latency). But the issue with paging is you have no idea when it’s gonna hit you, and may impact a job that’s behaving perfectly fine, except something got paged out by a badly behaving job. Ultimately disabling paging is a really good tool for limiting the blast radius of bad behaviour, and making cause-and-effect highly correlated (oh look thing X just OOMed, probably means thing X consumed too much memory. Rather than thing Y has strange tail latency because thing X keeps consuming too much memory). It’s failing fast, but for memory rather than exceptions.
- Galanwe 7y ago
  I get your point, and I think we agree here. I mainly wanted to argue against the original statement, which was quite broad “high performance machines running in a production environments will absolutely have swap disabled”. Would you agree to rephrase both our arguments by: - paging incurs seemingly random performance degradation of processes and should be avoided - if you have a form of task queue/job distribution system which handles automatic re-run, and can afford at no business cost to restart a process from scratch, then disabling swapping allows fail fast behaviour - otherwise swapping can be used as a safety net for programs that would be better off slightly late than restarted from scratch - both scenarios require sane monitoring of process behaviours, to catch symptomatic failures/restart in case 1) and recurring swap usage in case 2)
- AdamJacobMuller 7y ago
  > if your working under tight latency requirements then paging can push you over that boundary Even if you're not under tight requirements, swap can do strange things. I've actually seen situations where hitting swap, even trivially, can cause massive increases in latency. I'm talking about jobs which took 10s of milliseconds to complete now taking multiple 10s of seconds. I've even seem some absurdly bad memory management where Linux will make very very poor choices about what to page out. > Ultimately disabling paging is a really good tool for limiting the blast radius of bad behaviour 1000% agreed. Fail fast rather than fail slowly.
rkagerer 7y ago
>> “process failure would generally be compensated-for at a higher level” > I don’t quite get it. What does it mean? You re-run the process at an other time? On an other machine? I expect the commenter means whatever thing kicked off this job will detect it failed and do something about it (try it again elsewhere, log an alert triggering an admin to go provision more capacity, etc.) Their view is that a failure is easier to troubleshoot and fix than success with intermittently anomalous characteristics.
- Galanwe 7y ago
  Got it. To be honest, what I really wanted here is to force the commenter on agreeing that whatever the architecture/framework/etc., there is no magical solution, and saying that it will "be compensated-for at a higher level" just hides the only 2 real possibilities: - Kill the process and restart it somewhere else (no swap) - Wait for the process to finish anyway (using swap) Both choices have their own pros and cons, as discussed in other comments. But arguments such as "it's better to have predictable process termination", and "under tight latency requirements then paging can push you over that boundary" are exposed as much less meaningful once the problem is stated in simple terms. If the process indeed uses more memory than planned, then you have no guarantee than restarting it will work this time. Worst case, you can even have to allocate urgently a new node to handle that 1MB of memory above what was available. Not sure that is really the best solution "under tight latency requirements". My experience would indeed be the reverse: if you are faced with unexpected memory usage growth, then having some swap on the side to allow your process to finish makes for a much smoother production system than having to restart it all from scratch somewhere else/at an other time. In my experience, production is not the place where you want to enforce ideological concepts such as "a process shall not use more memory than what was planned". Production is the place where you make compromises and account for everything that slipped through these concepts, because you have no choice: it _has_ to run. Staging would be a good place for this though, as it's much less critical if something fails, and you have more time to fix and account for it. I really don't want to debug/profile/release/deploy a program in a hurry just because somehow it ended up using more memory than planned. (or convince infrastructure to deploy a new node type ASAP to handle that new memory constraint). Now one could argue that such unplanned memory consumption should have been caught earlier in the development process, to which I would reply: - Not everyone is able to properly test programs in a production like environment, especially when dealing with resource-hungry processes. In our case for instance a typical staging machine would be a downscaled version of the production one (less cores, less memory, less bandwidth). - I tend to plan _also_ for what should _not_ happen (because it will happen). Even with a perfect environment to calibrate programs before a release, there will be releases in which the memory usage was not accounted for. Guaranteed.
mimi89999 7y ago
I have `vm.swappiness = 1` for that. Who would want a job to be OOM killed?