5 ms·

Yep, even with no swap whatsoever, performance is completely trashed (talking even the mouse lags for 30+ seconds at a time) for like a solid 5+ minutes before

by throwaway2048 7y ago

Yep, even with no swap whatsoever, performance is completely trashed (talking even the mouse lags for 30+ seconds at a time) for like a solid 5+ minutes before the oomkiller triggers, with swap, you might as well just reboot because the system will take perhaps an hour to start responding. Linux is completely useless with ram that is almost full in a way that OSX and windows absolutely are not.

mort96 7y ago
Yeah, any time I end up in that situation I just hold the power button and prey to the journaling gods. It's a serious issue, I'm extremely glad it looks like there's actually some progress on fixing it.
- minitech 7y ago
  Enable SysRq and use SysRq+F. https://news.ycombinator.com/item?id=16561746 https://news.ycombinator.com/item?id=16561746
blattimwind 7y ago
Linux absolutely, positively, requires a decent chunk of free memory because the kernel's algorithms simply do not work and in my 15+ years of Linux experience they NEVER worked. If Linux starts thrashing, and it does so easily, then the box takes what surely is an uncomputable amount of time to recover. One way to make relatively sure that this is always the case is to use a userspace daemon like earlyoom. Though to stay fair all desktop OS'es behave badly when put under memory pressure, it's just that Linux is an order of magnitude or so worse.
- maximente 7y ago
  are any of the BSDs better in your experience?
  - myrandomcomment 7y ago
    Well my memory for BSD is not fully clear, but when I looked at this back in 2008 BSD handled it better then Linux. The king of sorting this was Solaris. It was rock solid. I would still argue that Solaris is better then Linux as a server, but it does not matter as Linux won. Also, I want my Amgia back :)
    - ComputerGuru 7y ago
      I only ditched the Solaris train for FreeBSD when all hope was gone after Schwartz’s reign came to an end. Solaris got so many things right architecturally that are damn hard to shoehorn in after the fact. Until today, I don’t know if any other *nix has sorted out ABI compatibility across architectures, which allowed running applications cross-compiled for another target to run against the memory-resident kernel without virtualization (only instruction emulation when needed). In practice, it meant that you could run “universal” x86 binaries against both the x86 and x86_64 kernels, even calling into drivers (!!), with guaranteed compatibility. The last I had checked, FreeBSD had made it a goal to unify all numeric values of IOCTLs across architectures, but I don’t believe they are there yet.
      - floatboth 7y ago
        Well, a 32-bit Linux binary on 64-bit FreeBSD can call into the GPU driver and render 3D, at least :) But I don't think anything is guaranteed for sure. 32-bit crap is not exactly a priority, haha. On the other hand, syscalls across architectures of the same bitness are exactly the same, minus the newer architectures just not having some historical abominations like sbrk. (IIRC, syscalls are different between even aarch64 and amd64 on Linux, which is just… why?!?)
        ComputerGuru 7y ago
        Hey, pretty sure I recognize your name from other FreeBSD-related comments you've made on here in the past that I found worthy of saving! We are stuck with needing to support x86 and prefer to do so as seamlessly as possible for the end user (call us old school like that); to that end we have a single bootable ISO image containing our appliance that needs to run on generic x86 hardware ranging from cutting edge to 15+ years old (I've posted about that with tons of accolades plus patches maximizing compatibility to the FreeBSD dev ML [some|a long] time ago but never heard back). We ported the software over from a very messy Linux-based implementation to a much cleaner architecture under FreeBSD 9 what I now realize can be considered a pretty long time ago. With that background out of the way, we had written a custom scfb-powered "guaranteed compatible" X display driver (back before there was an upstream scfb driver, although I am sorry to see even with FreeBSD 12.0 that xf86-video-scfb does not work on quite a number of hardware profiles) that would work so long as the framebuffer did (and it usually does). I remember running into an issue (under 10.0) where even basic FBIO_* ioctl defines had different values when compiled for AMD64 vs i686, so although our x86 Xorg module would run under x64 (with the i386 compatibility module loaded, iirc?), it would fail to work as the wrong driver routine was being called into. I was honestly surprised it didn't work oob, as these were very basic ioctl calls. Of course today with UEFI, signed drivers, and everything else, it's a miracle we haven't already been forced to distribute separate ISOs, but that's really only because we gave up on signed booting and simply require a legacy CSM for UEFI hardware, which is more steps but generally universally available; I don't think there's any real impetus behind getting UEFI support for x86, so we still boot the old-fashioned way. Getting completely off topic, honestly both Linux and FreeBSD are quite literally decades behind when it comes to being able to guarantee unaccelerated video output at {,e,s,x}vga resolutions without requiring hardware-specific drivers; I don't think I have ever seen a MS-DOS/Windows PC that wouldn't at least boot in VGA mode going back at least all the way to early 90s. We certainly spent no time agonizing over getting video to work when Microsoft used to allow us to license WinPE to power our appliance images. I have an unfinished patch somewhere for the vt VGA driver that adds support for the various ioctls, but it seems the world has unfortunately decided that going with hardware-specific (k)drm drivers is the way forward. We decided to bite the bullet and now ship with those too, but it's about the farthest thing from bulletproof compatibility and neither the vendors nor the reverse-engineered community driver devs really bother formally testing these modules and it unfortunately very much shows. And now nVidia has discontinued providing proprietary x86 drivers for FreeBSD and Linux (although they notably still do for Solaris, almost certainly because of the aforementioned ABI compatibility guarantees there), so we're probably going to need to split our legacy/i686 and the uefi/x64 images soon enough.
floatingatoll 7y ago
Does the time until oomkiller change dramatically on spinning metal versus SATA SSD versus NVMe, in a swap-off scenario?
brendangregg 7y ago
I don't think that's a fair comparison: Do you normally run OSX with the swap files completely disabled, or Windows with the pagefile completely disabled? That's what this bug is describing. I'd bet things get pretty nasty on OSX and Windows too, if you tried that. Perhaps the real bug is that Linux distros make it easy to run swapless.
- throwaway2048 7y ago
  The behavior is even worse if you leave swap enabled, as I already detailed in my post...
- Aaargh20318 7y ago
  Isn’t iOS basically a flavor of macOS that runs without swap ?
  - earenndil 7y ago
    Yes, but it's also a heavily integrated environment that aggressively quits background programs on memory pressure.
    - codedokode 7y ago
      That is the only working solution with HTML and JS based applications or apps using GC. Applications should save the state and quit (or be ready to quit) instead of using memory in backround mode.
    - Aaargh20318 7y ago
      But isn't that exactly what the linked article advocates Linux should also do ? Before quitting background applications it first sends them a request to free memory, in a well-behaved iOS program you use this to clean up your caches and ensure your don't use more RAM than you absolutely need. You should also suspend your state to disk when your app is backgrounded so you can just continue where you left off if your app is killed. Many macOS apps also do this, you can forcefully restart a Mac and after a reboot it'll restore your session to pretty much the exact state you left it in, including any open 'unsaved' files. Linux could implement a similar mechanism to signal apps to clean themselves up and maybe a 'save your state, you're about to get killed' signal.

mort96 7y ago

Yeah, any time I end up in that situation I just hold the power button and prey to the journaling gods. It's a serious issue, I'm extremely glad it looks like there's actually some progress on fixing it.

minitech 7y ago
Enable SysRq and use SysRq+F. https://news.ycombinator.com/item?id=16561746 https://news.ycombinator.com/item?id=16561746

blattimwind 7y ago

Linux absolutely, positively, requires a decent chunk of free memory because the kernel's algorithms simply do not work and in my 15+ years of Linux experience they NEVER worked. If Linux starts thrashing, and it does so easily, then the box takes what surely is an uncomputable amount of time to recover. One way to make relatively sure that this is always the case is to use a userspace daemon like earlyoom. Though to stay fair all desktop OS'es behave badly when put under memory pressure, it's just that Linux is an order of magnitude or so worse.

maximente 7y ago
are any of the BSDs better in your experience?
- myrandomcomment 7y ago
  Well my memory for BSD is not fully clear, but when I looked at this back in 2008 BSD handled it better then Linux. The king of sorting this was Solaris. It was rock solid. I would still argue that Solaris is better then Linux as a server, but it does not matter as Linux won. Also, I want my Amgia back :)
  - ComputerGuru 7y ago
    I only ditched the Solaris train for FreeBSD when all hope was gone after Schwartz’s reign came to an end. Solaris got so many things right architecturally that are damn hard to shoehorn in after the fact. Until today, I don’t know if any other *nix has sorted out ABI compatibility across architectures, which allowed running applications cross-compiled for another target to run against the memory-resident kernel without virtualization (only instruction emulation when needed). In practice, it meant that you could run “universal” x86 binaries against both the x86 and x86_64 kernels, even calling into drivers (!!), with guaranteed compatibility. The last I had checked, FreeBSD had made it a goal to unify all numeric values of IOCTLs across architectures, but I don’t believe they are there yet.
    - floatboth 7y ago
      Well, a 32-bit Linux binary on 64-bit FreeBSD can call into the GPU driver and render 3D, at least :) But I don't think anything is guaranteed for sure. 32-bit crap is not exactly a priority, haha. On the other hand, syscalls across architectures of the same bitness are exactly the same, minus the newer architectures just not having some historical abominations like sbrk. (IIRC, syscalls are different between even aarch64 and amd64 on Linux, which is just… why?!?)
      - ComputerGuru 7y ago
        Hey, pretty sure I recognize your name from other FreeBSD-related comments you've made on here in the past that I found worthy of saving! We are stuck with needing to support x86 and prefer to do so as seamlessly as possible for the end user (call us old school like that); to that end we have a single bootable ISO image containing our appliance that needs to run on generic x86 hardware ranging from cutting edge to 15+ years old (I've posted about that with tons of accolades plus patches maximizing compatibility to the FreeBSD dev ML [some|a long] time ago but never heard back). We ported the software over from a very messy Linux-based implementation to a much cleaner architecture under FreeBSD 9 what I now realize can be considered a pretty long time ago. With that background out of the way, we had written a custom scfb-powered "guaranteed compatible" X display driver (back before there was an upstream scfb driver, although I am sorry to see even with FreeBSD 12.0 that xf86-video-scfb does not work on quite a number of hardware profiles) that would work so long as the framebuffer did (and it usually does). I remember running into an issue (under 10.0) where even basic FBIO_* ioctl defines had different values when compiled for AMD64 vs i686, so although our x86 Xorg module would run under x64 (with the i386 compatibility module loaded, iirc?), it would fail to work as the wrong driver routine was being called into. I was honestly surprised it didn't work oob, as these were very basic ioctl calls. Of course today with UEFI, signed drivers, and everything else, it's a miracle we haven't already been forced to distribute separate ISOs, but that's really only because we gave up on signed booting and simply require a legacy CSM for UEFI hardware, which is more steps but generally universally available; I don't think there's any real impetus behind getting UEFI support for x86, so we still boot the old-fashioned way. Getting completely off topic, honestly both Linux and FreeBSD are quite literally decades behind when it comes to being able to guarantee unaccelerated video output at {,e,s,x}vga resolutions without requiring hardware-specific drivers; I don't think I have ever seen a MS-DOS/Windows PC that wouldn't at least boot in VGA mode going back at least all the way to early 90s. We certainly spent no time agonizing over getting video to work when Microsoft used to allow us to license WinPE to power our appliance images. I have an unfinished patch somewhere for the vt VGA driver that adds support for the various ioctls, but it seems the world has unfortunately decided that going with hardware-specific (k)drm drivers is the way forward. We decided to bite the bullet and now ship with those too, but it's about the farthest thing from bulletproof compatibility and neither the vendors nor the reverse-engineered community driver devs really bother formally testing these modules and it unfortunately very much shows. And now nVidia has discontinued providing proprietary x86 drivers for FreeBSD and Linux (although they notably still do for Solaris, almost certainly because of the aforementioned ABI compatibility guarantees there), so we're probably going to need to split our legacy/i686 and the uefi/x64 images soon enough.

floatingatoll 7y ago

Does the time until oomkiller change dramatically on spinning metal versus SATA SSD versus NVMe, in a swap-off scenario?

brendangregg 7y ago

I don't think that's a fair comparison: Do you normally run OSX with the swap files completely disabled, or Windows with the pagefile completely disabled? That's what this bug is describing. I'd bet things get pretty nasty on OSX and Windows too, if you tried that. Perhaps the real bug is that Linux distros make it easy to run swapless.

throwaway2048 7y ago
The behavior is even worse if you leave swap enabled, as I already detailed in my post...
Aaargh20318 7y ago
Isn’t iOS basically a flavor of macOS that runs without swap ?
- earenndil 7y ago
  Yes, but it's also a heavily integrated environment that aggressively quits background programs on memory pressure.
  - codedokode 7y ago
    That is the only working solution with HTML and JS based applications or apps using GC. Applications should save the state and quit (or be ready to quit) instead of using memory in backround mode.
  - Aaargh20318 7y ago
    But isn't that exactly what the linked article advocates Linux should also do ? Before quitting background applications it first sends them a request to free memory, in a well-behaved iOS program you use this to clean up your caches and ensure your don't use more RAM than you absolutely need. You should also suspend your state to disk when your app is backgrounded so you can just continue where you left off if your app is killed. Many macOS apps also do this, you can forcefully restart a Mac and after a reboot it'll restore your session to pretty much the exact state you left it in, including any open 'unsaved' files. Linux could implement a similar mechanism to signal apps to clean themselves up and maybe a 'save your state, you're about to get killed' signal.