4 ms·

>However, what I've never done is actually benchmarked the same workload on machines with no, some and loads of swap. However, I generally defer to Rachel, beca

>However, what I've never done is actually benchmarked the same workload on machines with no, some and loads of swap. However, I generally defer to Rachel, because Rachel has been there and been bitten by that before. I have. A well utilized machine is going to absolutely tank once it hits swap. Do you want to engineer your application to be able to cope with two radically different performance regimes, or do you simply want to ensure that your working set stays bounded?

zaphar 7y ago
I have too. I've also been in the opposite scenario where not having swap caused run away service kill and restart scenarios. Reading between the lines it looks like Rachel has too. Which is why she advises a "not too much" swap approach. As is usual the answer is not one or the other extreme it's much more nuanced than that.
- VectorLock 7y ago
  >I've also been in the opposite scenario where not having swap caused run away service kill and restart scenarios. Seems like a good time to just kill it and move on. "Cattle not pets."
  - zaphar 7y ago
    The funny thing about cattle is that they stampede. One host goes down the load increases on the others. They go down too. That cascades until they are all in a kill/restart cycle. I've seen lack of swap cause this. I'm betting Rachel has too.
chc 7y ago
If you ensure your working set stays bounded, having swap enabled (say, with swappiness 1) is not a big deal. If you're worried about the performance characteristics of swap, you've already conceded that you aren't confident in the bounds of your working set. So it doesn't seem like the relevant question is "Do we want to have well-bounded memory or use swap?" so much as "Can you tolerate some temporary slowness more easily than a server crash?"
andyjpb 7y ago
It's still not as simple as that. Swap is there to relieve memory allocation pressure. Memory allocation pressure is an incredibly dense concept but it basically means "if lots of people are asking for fresh pages, how quickly can I service them"? There are also other types of memory pressure. One is "if lots of people are reading and writing to pages, how quickly can I service them?" This depends on the state of the mapping for the virtual page being read or written. If that virtual page has an associated physical page then the answer is "not too slowly". If the physical page is in one of the caches then the answer is "more quickly". If the relevant part of the page is in a register then the answer is "in one clock cycle". On the other hand, if the read or write is associated with a page that needs to be mapped in, either from a disk file or a swap file, then the answer is "slowly". These types of pressure need to be balanced with each other. The idea isn't just to keep as much data in physical RAM as possible. The idea is to use the RAM as effectively as possible (perhaps by keeping as much relevant data in RAM as possible) whilst also being able to respond effectively to requests for fresh pages. In general, under pressure and load, the Linux kernel tries to keep a few MiB of ready-to-allocate pages. When it runs out of these it raids the page cache. It's reasonably quick to get (non-dirty) pages from the page cache because they can just be zero'd and handed out. The most difficult pages to reclaim are those that must be copied to swap before they are zero'd. So the kernel tries to minimise that. How does it do that? Well, it has some tricks: When processes load and run they allocate a bunch of pages. Sometimes, and not infrequently, they allocate and write to pages which are never accessed again! I see this on my laptop all the time. I never use even close to the physical amount of RAM I have in the machine. Right now I'm using 4,959MiB of 15,799MiB and it rarely surpasses 6 or 7 GiB. However, if I leave it running for any length of time (a number of days or weeks) then I start to see a little bit of swap getting allocated. I've currently been up about 98 days and right now I'm using 317MiB of swap. What's happened here is that the kernel has swapped out pages that it thinks are never going to be used again. That way, if memory allocation pressure suddenly increases it's got those physical pages ready to service those requests. If there was no swap, those pages would be unnecessarily pinned into physical memory even tho' they would never be used. Another commenter asked a question like "What's the difference between a machine with some memory and some swap and a machine with more memory?" Well, there are a few subtle differences, but this is one of them. The machine with swap will have a higher percentage of physical memory available and will be able to respond faster to larger allocations. Another difference is price. RAM is still expensive compared to disk. If you have a 16GiB box with a bit of swap then you have a 16GiB box that you can use for data that's actively being read and written. If you don't have the swap then you're paying for some of that RAM that gets written and never used. This doesn't matter so much in the small scale but when you have a few machines you want to be getting the most out of them (and what that really means is the subject of another post!). Swap space gets you more bang for your buck. So swap space is really about giving the kernel a mechanism to manage the different types of memory pressure without taking too many compromises on the different trade-offs. Even if you're never using all your RAM, swap will still get used so that the machine can respond optimally to as many types of memory activity as possible. One thing I'd really like to know is whether the number `free` gives me for "Swap used" is the amount of data that's in swap and nowhere else or whether it's the amount of swap space that's used even if the pages are also still in physical memory.

zaphar 7y ago

I have too. I've also been in the opposite scenario where not having swap caused run away service kill and restart scenarios. Reading between the lines it looks like Rachel has too. Which is why she advises a "not too much" swap approach. As is usual the answer is not one or the other extreme it's much more nuanced than that.

VectorLock 7y ago
>I've also been in the opposite scenario where not having swap caused run away service kill and restart scenarios. Seems like a good time to just kill it and move on. "Cattle not pets."
- zaphar 7y ago
  The funny thing about cattle is that they stampede. One host goes down the load increases on the others. They go down too. That cascades until they are all in a kill/restart cycle. I've seen lack of swap cause this. I'm betting Rachel has too.

chc 7y ago

If you ensure your working set stays bounded, having swap enabled (say, with swappiness 1) is not a big deal. If you're worried about the performance characteristics of swap, you've already conceded that you aren't confident in the bounds of your working set. So it doesn't seem like the relevant question is "Do we want to have well-bounded memory or use swap?" so much as "Can you tolerate some temporary slowness more easily than a server crash?"

andyjpb 7y ago

It's still not as simple as that. Swap is there to relieve memory allocation pressure. Memory allocation pressure is an incredibly dense concept but it basically means "if lots of people are asking for fresh pages, how quickly can I service them"? There are also other types of memory pressure. One is "if lots of people are reading and writing to pages, how quickly can I service them?" This depends on the state of the mapping for the virtual page being read or written. If that virtual page has an associated physical page then the answer is "not too slowly". If the physical page is in one of the caches then the answer is "more quickly". If the relevant part of the page is in a register then the answer is "in one clock cycle". On the other hand, if the read or write is associated with a page that needs to be mapped in, either from a disk file or a swap file, then the answer is "slowly". These types of pressure need to be balanced with each other. The idea isn't just to keep as much data in physical RAM as possible. The idea is to use the RAM as effectively as possible (perhaps by keeping as much relevant data in RAM as possible) whilst also being able to respond effectively to requests for fresh pages. In general, under pressure and load, the Linux kernel tries to keep a few MiB of ready-to-allocate pages. When it runs out of these it raids the page cache. It's reasonably quick to get (non-dirty) pages from the page cache because they can just be zero'd and handed out. The most difficult pages to reclaim are those that must be copied to swap before they are zero'd. So the kernel tries to minimise that. How does it do that? Well, it has some tricks: When processes load and run they allocate a bunch of pages. Sometimes, and not infrequently, they allocate and write to pages which are never accessed again! I see this on my laptop all the time. I never use even close to the physical amount of RAM I have in the machine. Right now I'm using 4,959MiB of 15,799MiB and it rarely surpasses 6 or 7 GiB. However, if I leave it running for any length of time (a number of days or weeks) then I start to see a little bit of swap getting allocated. I've currently been up about 98 days and right now I'm using 317MiB of swap. What's happened here is that the kernel has swapped out pages that it thinks are never going to be used again. That way, if memory allocation pressure suddenly increases it's got those physical pages ready to service those requests. If there was no swap, those pages would be unnecessarily pinned into physical memory even tho' they would never be used. Another commenter asked a question like "What's the difference between a machine with some memory and some swap and a machine with more memory?" Well, there are a few subtle differences, but this is one of them. The machine with swap will have a higher percentage of physical memory available and will be able to respond faster to larger allocations. Another difference is price. RAM is still expensive compared to disk. If you have a 16GiB box with a bit of swap then you have a 16GiB box that you can use for data that's actively being read and written. If you don't have the swap then you're paying for some of that RAM that gets written and never used. This doesn't matter so much in the small scale but when you have a few machines you want to be getting the most out of them (and what that really means is the subject of another post!). Swap space gets you more bang for your buck. So swap space is really about giving the kernel a mechanism to manage the different types of memory pressure without taking too many compromises on the different trade-offs. Even if you're never using all your RAM, swap will still get used so that the machine can respond optimally to as many types of memory activity as possible. One thing I'd really like to know is whether the number `free` gives me for "Swap used" is the amount of data that's in swap and nowhere else or whether it's the amount of swap space that's used even if the pages are also still in physical memory.