19 ms·

Removing swap is something most people do in production because performance matters. It's been this way for 20 years. Was that not normal?

by mannyv 4y ago

bityard 4y ago
Swap actually increases system performance for many common workloads: https://chrisdown.name/2018/01/02/in-defence-of-swap.html https://chrisdown.name/2018/01/02/in-defence-of-swap.html
- morelisp 4y ago
  The basic argument is that swap can be preferrable to hard eviction for managing disk cache, which can be true, but disk-heavy workloads are less common than ever especially in general-use clusters. (The other argument is the usual mix of "but it can swap out unused areas" in which case you should fix the program to not allocate a ton of memory it doesn't need; and "but the OOM killer will come" and yes that's the entire point please move my process elsewhere).
  - shawnz 4y ago
    > "but it can swap out unused areas" in which case you should fix the program to not allocate a ton of memory it doesn't need Isn't this basically asking every application to independently reimplement their own swap-style functionality? Surely it makes more sense to have swap as a system level feature that every application can take advantage of? That way the system has more information available to make swapping decisions since it can take into account the memory usage of all applications and drivers/etc together, and applications can take advantage of the most advanced swapping algorithms in the latest versions of the system without having to be individually updated.
    - morelisp 4y ago
      > Isn't this basically asking every application to independently reimplement their own swap-style functionality? No, it's asking for applications not to allocate a bunch of space they don't need in the first place.
      - shawnz 4y ago
        If you are suggesting that your applications are allocating a bunch of memory which they literally never use in any way, then I don't think that is really a common issue in practice. In practice I expect the typical scenario you're thinking of is applications are allocating a bunch of memory which they use only briefly or rarely and could easily fetch or recompute that data again later if needed. That's exactly the scenario which swap optimizes for. So why should every application individually implement logic to optimize for it? Fixing these kind of issues is not as simple as "just don't allocate the memory".
        morelisp 4y ago
        > If you are suggesting that your applications are allocating a bunch of memory which they literally never use in any way, then I don't think that is really a common issue in practice. That is what I'm suggesting, and I suggest you look at how much space is wasted by startup-initialized data in libraries for features you'll never use, JIT representations that never get compiled for more than one shape, class metadata that never gets touched and vtables which don't get used, how easy it is in any GC language to accidentally keep a large buffer alive until the end of a lexical scope rather than its last real use, etc. etc. Today's programs are bloated as hell dude.
        shawnz 4y ago
        All these problems you're describing are way more complicated than "just don't allocate the memory" and would involve sophisticated logic to optimize (like predicting exactly what features are or are not going to be used by an application at start time without significantly increasing the startup latency, etc). Why not just let swap be that optimization?
        morelisp 4y ago
        Mediocrity uber alles forever!
        4y ago
- hedora 4y ago
  Yes, but it helps for essentially zero large server workloads. (Which is why most big server deployments disable it.)
- nisa 4y ago
  This article is always thrown around if the topic comes up, at least in my experience it's wrong or at least not useful for my usecases (general purpose web hosting stuff in all kinds of flavours) - swap on spinning rust outright kills the system if memory is low. it's a full blown crash. - swap on ssd act's much the same - zram swap stalls and slows down the system for several minutes I just raise vm.min_free_kbytes to make the oom killer faster and disable swap. At least the machines don't crash hard this way.
  - AshamedCaptain 4y ago
    You are either abusing the word "crash" or otherwise make a very dubious assertion here.
    - nisa 4y ago
      crash as in it's not recoverable using ssh - machine only responds to ping and needs a hard reset.
      - andrewjf 4y ago
        Which is exactly the situation you never, ever want a server in. So it’s actually worse than a “crash” Me and my team removed swap on tens of thousands of machines cause we were sick of dealing with this. We wanted the machines to fail hard and fast and not go into a state where it’s doing only a minuscule amount of actual work while trying to recover.
        cosarara 4y ago
        And I think that, sadly, it's not even a full solution, because linux can manage to get thrashing even without swap. It pages in and out things like memory mapped files or the content of executables of stopped processes. See for instance: https://serverfault.com/questions/898388/how-to-prevent-kernel-i-o-thrashing-in-absence-of-swap-memory-on-linux https://serverfault.com/questions/898388/how-to-prevent-kern...
  - 4y ago
- throwaway09223 4y ago
  It really doesn't. When the author article writes about swap helping "Under moderate/high memory contention" -- most people have already powercycled their device at this point (or monitoring has pulled the struggling node from production). It's utterly useless, due to the huge difference in latency between RAM and storage. On most production systems, memory allocation is roughly 1% system and 99% application (not counting temporary, evictable allocations like disk cache). Modern applications are not designed to have their pages swapped out to disk and it is not particularly helpful to swap the teeny tiny bits of OS. There are parts of this article which are true, but the overall picture it paints is not accurate and the conclusion that swap is helpful is not correct for high performance systems - both workstations and servers.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
marginalia_nu 4y ago
Linux paging behavior without swap is a complete train wreck. Doesn't have to be big, but without it, you can bring the system to its knees with 30% ostensibly free ram.
_jal 4y ago
It is normal, at least in my world. I don't find the term "performance" useful outside of a specific context. The main reason we don't use swap is predictability. We know how much memory a particular VM should be using. If it exceeds that, failing quickly rather than changing behavior (slowing down) is far preferable, and then you correct whatever the problem is. All the testing we've done with swap has shown it to have negative in server environments. I see it as useful for client machines, and maybe a bandaid to get by with underpowered systems if you have to for some reason. But that shouldn't happen in prod, especially if you're a public cloud user, as most folks here seem to be.
- kevincox 4y ago
  I think that is where the confusion comes in. Almost everyone agrees that if your workload is slowing down due to paging that is bad. If your working set is bigger than your RAM you have a problem, swap or not. However swap helps optimize your RAM usage so that your working set can fit with less waste. It allows the kernel more flexibility with what pages to evict from RAM. Without swap it can only evict pages backed by files. If you start evicting files that are in your working set you are just as screwed as if it starts evicting anonymous pages in your working set. With swap it can evict other unused pages before touching the pages in your working set. I think there is some truth that it can be harder to notice with swap because there is more buffer between running great and literally crashing. However, in either situation you should be monitoring IO wait and application performance to ensure that you have enough memory for your working set.
shawnz 4y ago
Swap is a performance optimization, so why would you remove it if you care about performance? Contrary to popular belief, swap isn't a way to get extra memory for free: it won't help you if your workload size is bigger than your physical memory capacity and that is not what it is meant to be used for.
- kevin_thibedeau 4y ago
  It was an optimization when RAM was too expensive to keep a full working set live.
  - shawnz 4y ago
    No, that's exactly what I'm saying: swap was never meant to let you run workloads bigger than your physical memory capacity.
    - deleted 4y ago
      [deleted]

Removing swap is something most people do in production because performance matters. It's been this way for 20 years. Was that not normal?

by mannyv 4y ago

Removing swap is something most people do in production because performance matters. It's been this way for 20 years. Was that not normal?

bityard 4y ago

Swap actually increases system performance for many common workloads: https://chrisdown.name/2018/01/02/in-defence-of-swap.html https://chrisdown.name/2018/01/02/in-defence-of-swap.html

morelisp 4y ago
The basic argument is that swap can be preferrable to hard eviction for managing disk cache, which can be true, but disk-heavy workloads are less common than ever especially in general-use clusters. (The other argument is the usual mix of "but it can swap out unused areas" in which case you should fix the program to not allocate a ton of memory it doesn't need; and "but the OOM killer will come" and yes that's the entire point please move my process elsewhere).
- shawnz 4y ago
  > "but it can swap out unused areas" in which case you should fix the program to not allocate a ton of memory it doesn't need Isn't this basically asking every application to independently reimplement their own swap-style functionality? Surely it makes more sense to have swap as a system level feature that every application can take advantage of? That way the system has more information available to make swapping decisions since it can take into account the memory usage of all applications and drivers/etc together, and applications can take advantage of the most advanced swapping algorithms in the latest versions of the system without having to be individually updated.
  - morelisp 4y ago
    > Isn't this basically asking every application to independently reimplement their own swap-style functionality? No, it's asking for applications not to allocate a bunch of space they don't need in the first place.
    - shawnz 4y ago
      If you are suggesting that your applications are allocating a bunch of memory which they literally never use in any way, then I don't think that is really a common issue in practice. In practice I expect the typical scenario you're thinking of is applications are allocating a bunch of memory which they use only briefly or rarely and could easily fetch or recompute that data again later if needed. That's exactly the scenario which swap optimizes for. So why should every application individually implement logic to optimize for it? Fixing these kind of issues is not as simple as "just don't allocate the memory".
      - morelisp 4y ago
        > If you are suggesting that your applications are allocating a bunch of memory which they literally never use in any way, then I don't think that is really a common issue in practice. That is what I'm suggesting, and I suggest you look at how much space is wasted by startup-initialized data in libraries for features you'll never use, JIT representations that never get compiled for more than one shape, class metadata that never gets touched and vtables which don't get used, how easy it is in any GC language to accidentally keep a large buffer alive until the end of a lexical scope rather than its last real use, etc. etc. Today's programs are bloated as hell dude.
        shawnz 4y ago
        All these problems you're describing are way more complicated than "just don't allocate the memory" and would involve sophisticated logic to optimize (like predicting exactly what features are or are not going to be used by an application at start time without significantly increasing the startup latency, etc). Why not just let swap be that optimization?
        morelisp 4y ago
        Mediocrity uber alles forever!
        4y ago
hedora 4y ago
Yes, but it helps for essentially zero large server workloads. (Which is why most big server deployments disable it.)
nisa 4y ago
This article is always thrown around if the topic comes up, at least in my experience it's wrong or at least not useful for my usecases (general purpose web hosting stuff in all kinds of flavours) - swap on spinning rust outright kills the system if memory is low. it's a full blown crash. - swap on ssd act's much the same - zram swap stalls and slows down the system for several minutes I just raise vm.min_free_kbytes to make the oom killer faster and disable swap. At least the machines don't crash hard this way.
- AshamedCaptain 4y ago
  You are either abusing the word "crash" or otherwise make a very dubious assertion here.
  - nisa 4y ago
    crash as in it's not recoverable using ssh - machine only responds to ping and needs a hard reset.
    - andrewjf 4y ago
      Which is exactly the situation you never, ever want a server in. So it’s actually worse than a “crash” Me and my team removed swap on tens of thousands of machines cause we were sick of dealing with this. We wanted the machines to fail hard and fast and not go into a state where it’s doing only a minuscule amount of actual work while trying to recover.
      - cosarara 4y ago
        And I think that, sadly, it's not even a full solution, because linux can manage to get thrashing even without swap. It pages in and out things like memory mapped files or the content of executables of stopped processes. See for instance: https://serverfault.com/questions/898388/how-to-prevent-kernel-i-o-thrashing-in-absence-of-swap-memory-on-linux https://serverfault.com/questions/898388/how-to-prevent-kern...
- 4y ago
throwaway09223 4y ago
It really doesn't. When the author article writes about swap helping "Under moderate/high memory contention" -- most people have already powercycled their device at this point (or monitoring has pulled the struggling node from production). It's utterly useless, due to the huge difference in latency between RAM and storage. On most production systems, memory allocation is roughly 1% system and 99% application (not counting temporary, evictable allocations like disk cache). Modern applications are not designed to have their pages swapped out to disk and it is not particularly helpful to swap the teeny tiny bits of OS. There are parts of this article which are true, but the overall picture it paints is not accurate and the conclusion that swap is helpful is not correct for high performance systems - both workstations and servers.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

marginalia_nu 4y ago

Linux paging behavior without swap is a complete train wreck. Doesn't have to be big, but without it, you can bring the system to its knees with 30% ostensibly free ram.

_jal 4y ago

It is normal, at least in my world. I don't find the term "performance" useful outside of a specific context. The main reason we don't use swap is predictability. We know how much memory a particular VM should be using. If it exceeds that, failing quickly rather than changing behavior (slowing down) is far preferable, and then you correct whatever the problem is. All the testing we've done with swap has shown it to have negative in server environments. I see it as useful for client machines, and maybe a bandaid to get by with underpowered systems if you have to for some reason. But that shouldn't happen in prod, especially if you're a public cloud user, as most folks here seem to be.

kevincox 4y ago
I think that is where the confusion comes in. Almost everyone agrees that if your workload is slowing down due to paging that is bad. If your working set is bigger than your RAM you have a problem, swap or not. However swap helps optimize your RAM usage so that your working set can fit with less waste. It allows the kernel more flexibility with what pages to evict from RAM. Without swap it can only evict pages backed by files. If you start evicting files that are in your working set you are just as screwed as if it starts evicting anonymous pages in your working set. With swap it can evict other unused pages before touching the pages in your working set. I think there is some truth that it can be harder to notice with swap because there is more buffer between running great and literally crashing. However, in either situation you should be monitoring IO wait and application performance to ensure that you have enough memory for your working set.

shawnz 4y ago

Swap is a performance optimization, so why would you remove it if you care about performance? Contrary to popular belief, swap isn't a way to get extra memory for free: it won't help you if your workload size is bigger than your physical memory capacity and that is not what it is meant to be used for.

kevin_thibedeau 4y ago
It was an optimization when RAM was too expensive to keep a full working set live.
- shawnz 4y ago
  No, that's exactly what I'm saying: swap was never meant to let you run workloads bigger than your physical memory capacity.
  - deleted 4y ago
    [deleted]