6 ms·

Further to the comments about the pager hammering the disk to read clean pages (mainly but not exclusively binaries) even if swapping is disabled: In many cases

Further to the comments about the pager hammering the disk to read clean pages (mainly but not exclusively binaries) even if swapping is disabled: In many cases adding swap space will reduce the amount of paging which occurs. Many long-lived processes are completely idle (when was the last time that `getty ttyv6` woke up?) or at a minimum have pages of memory which are never used (e.g. the bottom page of main's stack). Evicting these "theoretically accessible but in practice never accessed" pages for memory frees up more memory for the things which matter.

throwaway3627 7y ago
Linux resource scheduling and prioritization and is pretty awful compared to its popularity. TBH, there are very few OSes that get high-pressure resource scheduling and prioritization right under nearly all normal circumstances. The hackaround for decades on Linux is always adding a tiny swap device, say 64-256 MiB on a fast device in order to 0) detect average high memory pressure with monitoring tools 1) prevent oddities under load without swap (as in OP example).
- ddingus 7y ago
  Sgi IRIX nailed this, FWIW. I would have thought some of the IRIX scheduler made it into Linux by now.
  - Annatar 7y ago
    No way, any sufficiently advanced technology is indistinguishable from magic and IRIX was so very, very advanced. IRIX hasn't been in development for almost two decades now and it's still more advanced in aspects like guaranteed I/O and software management (inst(1M))... What does that say about it and what does it say about the engineers who worked on it?
    - twic 7y ago
      XFS is still better than any version of that ext dreck!
      - vthriller 7y ago
        That's debatable. Better at losing file contents after hard reset? Maybe.
    - ddingus 7y ago
      Makes me want to gather them in a room and continue the process. Irix was so very good.
    - ddingus 7y ago
      And you are right on all counts. Inst was magic. The things I did, often on live systems...
  - IcePic 7y ago
    Irix even had "virtual swap" which had no (or very insufficient sized) backing store for it, just to handle all the superlarge allocations from which it only uses a tiny amount.
- nathell 7y ago
  As crazy as it sounds, can that fast device be a ramdisk?
  - GuB-42 7y ago
    Exactly what I thought. Losing <5% of your RAM as swap space probably won't do much difference. But if it solves problems, why doesn't the kernel automatically assign part of RAM as a virtual swap device, if the user allows it. It can help monitoring tools, and because the kernel knows it is not a real swap device, it can optimize its use.
  - LargoLasskhyfv 7y ago
    Not crazy at all, very useful. https://en.wikipedia.org/wiki/Zram https://en.wikipedia.org/wiki/Zram https://wiki.gentoo.org/wiki/Zram https://wiki.gentoo.org/wiki/Zram https://wiki.debian.org/ZRam https://wiki.debian.org/ZRam Or Armbian, a Debian derivative for many RaspberryPi-like ARM boards, where it helps avoiding trashing the usual sd-cards, and holds the logs, with different compression algorithms for /var/log, /tmp and swap. Their script is here: https://github.com/armbian/build/blob/master/packages/bsp/common/usr/lib/armbian/armbian-zram-config https://github.com/armbian/build/blob/master/packages/bsp/co... I really like how this enables a tiny NanoPi Neo2 with one GB Ram booting from a 64GB SD-Card in an aluminum case with SATA-adapter and a 2,5" HDD mostly idling to draw only about 1W from the wall socket, while clocking up from about 400Mhz to 1,3Ghz if it needs to. It's not the fastest, but sufficient.
  - pantalaimon 7y ago
    It's not that crazy, there is zram: https://www.kernel.org/doc/Documentation/blockdev/zram.txt https://www.kernel.org/doc/Documentation/blockdev/zram.txt
quazeekotl 7y ago
Unfortunately enabling swap in linux has a very annoying side effect, linux will preferentially push out pages of running programs that have been untouched for X time for more disk cache, pretty much no matter how much ram you have. This comes into play when you copy or access huge files that are going to be read exactly once, they will start pushing out untouched program pages to disk, in exchange for disk cache that is completely 100% useless, even to the tune of hundreds of gigabytes of it. Programs can reduce the problem with madvise(MADV_DONT_NEED), but that only applies to files you are mmap()ing, and every single program under the sun needs to be patched to issue these calls. You can adjust vm.swapiness systctl to make X larger, but no matter what, programs will start to get pushed out to disk eventually, and cause unresponsiveness when activated. You can reduce vm.swapiness to 1, but if you do, the system only starts swapping in an absolute critical low ram situation and you encounter anywhere from 5 minutes, to 1+ hour of total, complete unresponsiveness in a low ram situation. There _NEEDS_ to be a setting where program pages don't get pushed out for disk cache, peroid, unless approaching a low ram situation, but BEFORE it causes long periods of total crushing unresponsiveness.
- the8472 7y ago
  > but that only applies to files you are mmap()ing fadvise provides the same for file descriptors. some tools such as rsync make use of it to prevent clobbering the page cache when streaming files.
  - quazeekotl 7y ago
    nice! was not aware of that syscall, however, patching the entire world remains...
    - compsciphd 7y ago
      it might be possible to create an LD_PRELOAD'd library that wraps open type syscalls (i.e. those that return an fd, might just be open, haven't kept up with all of linux syscalls) and that based on a config file calls fadvise on those fd's that correspond to specific files/paths on disk). Won't help for statically linked binaries or those that call syscalls directly without glibc's shims, but that should be a small number of programs. heck, if I were still a phd student, I'd want to run performance numbers on this in many different scenarios and see how performance behaves. feel like there could be a paper here.
mnw21cam 7y ago
Yes, this is exactly the cause. The pager hammers the disc to read clean pages, because they don't count as swap. And I agree that a small amount of swap can actually reduce the paging that occurs, if you assume that the amount of RAM required is independent of the amount of RAM available. However, as we all know, stuff grows to fill the available space, and if you do configure swap you just delay the inevitable, not prevent it. Having said that, having swap available means that when memory pressure occurs, you have a more graceful degradation in service, because the first you know about it is when the kernel starts swapping out an idle process to free memory for the new memory hog that you are interacting with. This slows down your interactive session, but not as much as if you have no swap available - in that case, the system suddenly and drastically reduces in performance because it is trying to swap in your interactive process. The more graceful degradation of having some swap available gives you a chance to realise that you are doing more than your computer can cope with, and stop. As far as I see it, there are three solutions: 1. Disable overcommit. This tends to not play very nicely with things that allocate loads of virtual memory and not use it, like Java. And if you do have a load of processes that actually use all the memory they allocated, then you can still have the same problem occur. The solution to that one is to get the kernel to say no early, before the system actually runs out of RAM. 2. Get the OOM killer to kill things much earlier, before the system starts throwing away clean pages that are actually being used. On my system with 384GB RAM, I have installed earlyoom, and instructed it to trigger the OOM killer if the free RAM falls below 10% (and remember that stuff you are actually using, but happens to be clean, counts as free RAM). This is the easiest and quickest solution right now. If your main objection to this is that you are inviting the system to kill things that might be important, remember that the kernel already does this, and if you don't like it you should use option 1 above (and really hope that all your software handles malloc failure correctly). 3. Introduce a new system in the kernel to mark pages that are actually being regularly used but are clean as "important", and no longer count as free RAM for the purposes of calculating memory pressure. This could either be as a new madvise (but it would be impractical to get all software developers to start using this), or by marking all binary text by default (which would neglect the large read-only databases that some programs hammer), or by some heuristics. This would then trigger the OOM killer (or the allocator to say no, depending on overcommit) when actual free RAM is low.
snvzz 7y ago
Swap has a side effect that's not very nice: It makes memory non-deterministic, as disk is non-deterministic. Linux does unfortunately have serious issues to do with peaks of latency which make it behave horribly with realtime tasks such as pro audio. It's so bad that it's often perceiveable in desktop usage. linux-rt does mitigate this considerably, but it's still not very good. I'm hopeful for genode (with seL4 specifically), haiku, helenos and fuchsia.

throwaway3627 7y ago

Linux resource scheduling and prioritization and is pretty awful compared to its popularity. TBH, there are very few OSes that get high-pressure resource scheduling and prioritization right under nearly all normal circumstances. The hackaround for decades on Linux is always adding a tiny swap device, say 64-256 MiB on a fast device in order to 0) detect average high memory pressure with monitoring tools 1) prevent oddities under load without swap (as in OP example).

ddingus 7y ago
Sgi IRIX nailed this, FWIW. I would have thought some of the IRIX scheduler made it into Linux by now.
- Annatar 7y ago
  No way, any sufficiently advanced technology is indistinguishable from magic and IRIX was so very, very advanced. IRIX hasn't been in development for almost two decades now and it's still more advanced in aspects like guaranteed I/O and software management (inst(1M))... What does that say about it and what does it say about the engineers who worked on it?
  - twic 7y ago
    XFS is still better than any version of that ext dreck!
    - vthriller 7y ago
      That's debatable. Better at losing file contents after hard reset? Maybe.
  - ddingus 7y ago
    Makes me want to gather them in a room and continue the process. Irix was so very good.
  - ddingus 7y ago
    And you are right on all counts. Inst was magic. The things I did, often on live systems...
- IcePic 7y ago
  Irix even had "virtual swap" which had no (or very insufficient sized) backing store for it, just to handle all the superlarge allocations from which it only uses a tiny amount.
nathell 7y ago
As crazy as it sounds, can that fast device be a ramdisk?
- GuB-42 7y ago
  Exactly what I thought. Losing <5% of your RAM as swap space probably won't do much difference. But if it solves problems, why doesn't the kernel automatically assign part of RAM as a virtual swap device, if the user allows it. It can help monitoring tools, and because the kernel knows it is not a real swap device, it can optimize its use.
- LargoLasskhyfv 7y ago
  Not crazy at all, very useful. https://en.wikipedia.org/wiki/Zram https://en.wikipedia.org/wiki/Zram https://wiki.gentoo.org/wiki/Zram https://wiki.gentoo.org/wiki/Zram https://wiki.debian.org/ZRam https://wiki.debian.org/ZRam Or Armbian, a Debian derivative for many RaspberryPi-like ARM boards, where it helps avoiding trashing the usual sd-cards, and holds the logs, with different compression algorithms for /var/log, /tmp and swap. Their script is here: https://github.com/armbian/build/blob/master/packages/bsp/common/usr/lib/armbian/armbian-zram-config https://github.com/armbian/build/blob/master/packages/bsp/co... I really like how this enables a tiny NanoPi Neo2 with one GB Ram booting from a 64GB SD-Card in an aluminum case with SATA-adapter and a 2,5" HDD mostly idling to draw only about 1W from the wall socket, while clocking up from about 400Mhz to 1,3Ghz if it needs to. It's not the fastest, but sufficient.
- pantalaimon 7y ago
  It's not that crazy, there is zram: https://www.kernel.org/doc/Documentation/blockdev/zram.txt https://www.kernel.org/doc/Documentation/blockdev/zram.txt

quazeekotl 7y ago

Unfortunately enabling swap in linux has a very annoying side effect, linux will preferentially push out pages of running programs that have been untouched for X time for more disk cache, pretty much no matter how much ram you have. This comes into play when you copy or access huge files that are going to be read exactly once, they will start pushing out untouched program pages to disk, in exchange for disk cache that is completely 100% useless, even to the tune of hundreds of gigabytes of it. Programs can reduce the problem with madvise(MADV_DONT_NEED), but that only applies to files you are mmap()ing, and every single program under the sun needs to be patched to issue these calls. You can adjust vm.swapiness systctl to make X larger, but no matter what, programs will start to get pushed out to disk eventually, and cause unresponsiveness when activated. You can reduce vm.swapiness to 1, but if you do, the system only starts swapping in an absolute critical low ram situation and you encounter anywhere from 5 minutes, to 1+ hour of total, complete unresponsiveness in a low ram situation. There _NEEDS_ to be a setting where program pages don't get pushed out for disk cache, peroid, unless approaching a low ram situation, but BEFORE it causes long periods of total crushing unresponsiveness.

the8472 7y ago
> but that only applies to files you are mmap()ing fadvise provides the same for file descriptors. some tools such as rsync make use of it to prevent clobbering the page cache when streaming files.
- quazeekotl 7y ago
  nice! was not aware of that syscall, however, patching the entire world remains...
  - compsciphd 7y ago
    it might be possible to create an LD_PRELOAD'd library that wraps open type syscalls (i.e. those that return an fd, might just be open, haven't kept up with all of linux syscalls) and that based on a config file calls fadvise on those fd's that correspond to specific files/paths on disk). Won't help for statically linked binaries or those that call syscalls directly without glibc's shims, but that should be a small number of programs. heck, if I were still a phd student, I'd want to run performance numbers on this in many different scenarios and see how performance behaves. feel like there could be a paper here.

mnw21cam 7y ago

Yes, this is exactly the cause. The pager hammers the disc to read clean pages, because they don't count as swap. And I agree that a small amount of swap can actually reduce the paging that occurs, if you assume that the amount of RAM required is independent of the amount of RAM available. However, as we all know, stuff grows to fill the available space, and if you do configure swap you just delay the inevitable, not prevent it. Having said that, having swap available means that when memory pressure occurs, you have a more graceful degradation in service, because the first you know about it is when the kernel starts swapping out an idle process to free memory for the new memory hog that you are interacting with. This slows down your interactive session, but not as much as if you have no swap available - in that case, the system suddenly and drastically reduces in performance because it is trying to swap in your interactive process. The more graceful degradation of having some swap available gives you a chance to realise that you are doing more than your computer can cope with, and stop. As far as I see it, there are three solutions: 1. Disable overcommit. This tends to not play very nicely with things that allocate loads of virtual memory and not use it, like Java. And if you do have a load of processes that actually use all the memory they allocated, then you can still have the same problem occur. The solution to that one is to get the kernel to say no early, before the system actually runs out of RAM. 2. Get the OOM killer to kill things much earlier, before the system starts throwing away clean pages that are actually being used. On my system with 384GB RAM, I have installed earlyoom, and instructed it to trigger the OOM killer if the free RAM falls below 10% (and remember that stuff you are actually using, but happens to be clean, counts as free RAM). This is the easiest and quickest solution right now. If your main objection to this is that you are inviting the system to kill things that might be important, remember that the kernel already does this, and if you don't like it you should use option 1 above (and really hope that all your software handles malloc failure correctly). 3. Introduce a new system in the kernel to mark pages that are actually being regularly used but are clean as "important", and no longer count as free RAM for the purposes of calculating memory pressure. This could either be as a new madvise (but it would be impractical to get all software developers to start using this), or by marking all binary text by default (which would neglect the large read-only databases that some programs hammer), or by some heuristics. This would then trigger the OOM killer (or the allocator to say no, depending on overcommit) when actual free RAM is low.

snvzz 7y ago

Swap has a side effect that's not very nice: It makes memory non-deterministic, as disk is non-deterministic. Linux does unfortunately have serious issues to do with peaks of latency which make it behave horribly with realtime tasks such as pro audio. It's so bad that it's often perceiveable in desktop usage. linux-rt does mitigate this considerably, but it's still not very good. I'm hopeful for genode (with seL4 specifically), haiku, helenos and fuchsia.