6 ms·

I agree with the author's overall premise, which is that other people who don't own and who are not responsible for your system should not have authority over h

I agree with the author's overall premise, which is that other people who don't own and who are not responsible for your system should not have authority over how you build and operate it. One of the things that I like about the current company where I work is that owners always have the final say about their systems, not any external technical authority. There is no such authority who could say no where I work (with rare and limited exceptions), and I'm glad for that; service owners make the judgment call about trade-offs. > "so-and-so says no swap in production". That said, while I understand the point of the article is something different, I broadly agree with this advice: generally no swap in production. If you need swap, then that's probably a sign that the workload isn't matched well to the resources of the machine (unless the workload was specifically designed to employ swap, which most are not). The problem is that: maybe most of the time the application's working set will fit within memory, and so it will perform well, but some of the time it will begin to swap and performance will degrade. The result of adding swap to a system is non-deterministic performance. If you're building high-availability, high-performance production systems, then you don't want the potentiality of arbitrary performance degradation or non-deterministic behavior. Instead, you would prefer to solve the root cause -- such as by increasing the amount of memory available to the process or system, so that swap is no longer required. An engineer adding swap to a system is like a doctor treating a patient's symptoms instead of curing their disease. Both are useful but one is preferable. If you encounter a situation where you could solve an application problem by adding swap, then an even better solution is to address the root cause -- ensure the application doesn't run out of physical memory. Swap is a band-aid that masks a risk better addressed directly. Hence: no swap in production. (On the other hand, it might be fine to employ swap in production for services that aren't performance sensitive, and where it's preferable to accept the performance hit from using swap rather than pay more for better hardware. That's a fine trade-off if consciously made. In my experience, variations in performance are usually much more problematic to a business than changes to cost.)

_visgean 7y ago
But not having swap will cause the machine to misbehave (https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConsequence https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConseque...). Swap in this situation acts like a backup, surely you would prefer temporal degradation then total system shutdown. The solution here is continuous monitoring of free memory (we use datadog) + swap.
- terlisimo 7y ago
  > surely you would prefer temporal degradation then total system shutdown Sometimes, but sometimes you want the opposite. A slow service (e.g. a db server with a failing disk) might still pass health checks but degrade overall system performance. It is easier/simpler to detect a total outage than a slow/flapping service.
volfyd 7y ago
I think what may be happening here is that at some point the Linux kernel's behavior has changed and things run more smoothly with swap than without, even if you aren't swapping appreciably. There might be some code path it hits when it needs to flush cache or something like that that causes issue.
nwallin 7y ago
> generally no swap in production. This is a bad policy. The problem is that it breaks Linux's overcommit logic. Many applications allocate huge chunks of virtual memory far in excess of what they actually use. Due to the magic of virtual memory, this doesn't get mapped to physical memory until it's actually used. The trouble is that there's no mechanism in any programming language I'm aware of that allows the OS to retcon an allocation. The only tool the kernel has at its disposal is the oom killer. So if you allocate too much, and you have swap disabled, the oom killer has to step in and kill something even if you're only using half your ram. This is bad enough, but in addition to this, you have vfs cache to worry about. If you have 64GB ram, your applications are using 40GB, and you have 22GB cache, you and I might assume you have 24GB free, but the OS only sees 2GB free, and a few allocations might trigger the oom killer if they happen faster than the vfs can dump cache. (which is surprisingly slow) With swap, this isn't a big deal, because the pages don't actually get assigned until they are used, which happens later, so the vfs has plenty of time to evict. The point of swap isn't to give you more memory. It's so the OS can keep its promises about memory allocation and overcommitting.
- nailer 7y ago
  > So if you allocate too much, and you have swap disabled, the oom killer has to step in and kill something even if you're only using half your ram. You can allocate all the memory you want, the OOM killer only steps in when you actually use all the the memory.

_visgean 7y ago

But not having swap will cause the machine to misbehave (https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConsequence https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConseque...). Swap in this situation acts like a backup, surely you would prefer temporal degradation then total system shutdown. The solution here is continuous monitoring of free memory (we use datadog) + swap.

terlisimo 7y ago
> surely you would prefer temporal degradation then total system shutdown Sometimes, but sometimes you want the opposite. A slow service (e.g. a db server with a failing disk) might still pass health checks but degrade overall system performance. It is easier/simpler to detect a total outage than a slow/flapping service.

volfyd 7y ago

I think what may be happening here is that at some point the Linux kernel's behavior has changed and things run more smoothly with swap than without, even if you aren't swapping appreciably. There might be some code path it hits when it needs to flush cache or something like that that causes issue.

nwallin 7y ago

> generally no swap in production. This is a bad policy. The problem is that it breaks Linux's overcommit logic. Many applications allocate huge chunks of virtual memory far in excess of what they actually use. Due to the magic of virtual memory, this doesn't get mapped to physical memory until it's actually used. The trouble is that there's no mechanism in any programming language I'm aware of that allows the OS to retcon an allocation. The only tool the kernel has at its disposal is the oom killer. So if you allocate too much, and you have swap disabled, the oom killer has to step in and kill something even if you're only using half your ram. This is bad enough, but in addition to this, you have vfs cache to worry about. If you have 64GB ram, your applications are using 40GB, and you have 22GB cache, you and I might assume you have 24GB free, but the OS only sees 2GB free, and a few allocations might trigger the oom killer if they happen faster than the vfs can dump cache. (which is surprisingly slow) With swap, this isn't a big deal, because the pages don't actually get assigned until they are used, which happens later, so the vfs has plenty of time to evict. The point of swap isn't to give you more memory. It's so the OS can keep its promises about memory allocation and overcommitting.

nailer 7y ago
> So if you allocate too much, and you have swap disabled, the oom killer has to step in and kill something even if you're only using half your ram. You can allocate all the memory you want, the OOM killer only steps in when you actually use all the the memory.