4 ms·

I used to run a Linux workstation in the late 00's (sorry FreeBSD folks, I know, the shame...), and I ran a closed source PVR application on it. The memory acc

I used to run a Linux workstation in the late 00's (sorry FreeBSD folks, I know, the shame...), and I ran a closed source PVR application on it. The memory access pattern was pretty much pessimal for my use of the box as a workstation. I'd use it from 7am -> 8/9pm every day, then when I'd walk away from the keyboard, I'd watch HD recordings (which could be 7GB or more per hour). Those would get cached in memory, and eventually my workstation stuff (emacs, xterms, firefox, thunderbird) would start to get paged out. In the mornings, it was painful to start using each application, as it waited forever to page in from a spinning disk. I eventually wrote an LD_PRELOAD for the DVR software that overloaded open, and added O_DIRECT (to tell the kernel not to cache the data). This totally solved my problem, and didn't impact my DVR usage at all.

toast0 3y ago
> I used to run a Linux workstation in the late 00's (sorry FreeBSD folks, I know, the shame...), and I ran a closed source PVR application on it. It's ok, no shame. But as I understand it, FreeBSD would prefer to throw out (clean) disk cache pages under memory pressure until somewhere around FreeBSD 11 +/- 1, where there were a few changes that combined to make things like you described likely to happen. Heavy I/O overnight might still have been enough, and I'm not going to test run an old OS version to check ;) I can't find the changes quickly, but IIRC, older FreeBSD didn't mark anonymous pages as inactive unless there was heavy memory pressure; when there was mild memory pressure, it would go through the page queue(s) and free clean disk pages and skip other page; only taking action on a second pass if the first pass didn't clean enough. This usually meant your program pages would stay in memory, but when you hit memory pressure, there would be a big pause to mark a lot of pages inactive, often too many pages, which would then get faulted back to active... Current FreeBSD marks pages inactive on a more consistent basis, which is nice because when there is memory pressure, chancws are there's already classified pages. But it can lead to anonymous pages getting swapped out in favor of disk pages as you described; it's all tunable, of course, but it was a kind of weird transition for me. After upgrading the OS, some of my heavy i/o machines saw rising swap usage running the same software as before; took a while to figure that out.
- tonyarkles 3y ago
  Ran into an issue like this on relatively modern embedded Linux platform. It was a driver bug and it’s been fixed, but here’s the scenario: - heavy disk access because we were writing real-time images to an SSD at about 500MB/s - our application was steady-state about 4GB of RAM and we had 32GB available on the platform - the serial port that we received data from was, under the hood, using DMA In certain cases, Linux would completely run out of free pages (28GB of it being used for cache on files we were never going to read again). These were all available pages but just occupied at the exact moment. The serial driver would request a page for DMA when it received an interrupt and being inside an interrupt context would request that page with NOBLOCK. That meant that kmalloc would return NULL instead of giving a page, since it would need to evict one of the cache pages before one was available. The serial driver would then blow up and never retry the DMA transaction. Fun to debug that one!
  - toast0 3y ago
    DMA pages are fun. Some devices have special needs for DMA buffers, so maybe you've got something ancient that can only use memory under 32-bit, or maybe you have something really ancient that can only use memory under 16 MB; or maybe the disk controller is fine for regular disk access, but administrative commands need to use a limited range. I didn't really finish debugging that one, I got close enough and said well --- we can just measure SMART status a lot less frequently on boxes with that controller and called it a day. :)

toast0 3y ago

> I used to run a Linux workstation in the late 00's (sorry FreeBSD folks, I know, the shame...), and I ran a closed source PVR application on it. It's ok, no shame. But as I understand it, FreeBSD would prefer to throw out (clean) disk cache pages under memory pressure until somewhere around FreeBSD 11 +/- 1, where there were a few changes that combined to make things like you described likely to happen. Heavy I/O overnight might still have been enough, and I'm not going to test run an old OS version to check ;) I can't find the changes quickly, but IIRC, older FreeBSD didn't mark anonymous pages as inactive unless there was heavy memory pressure; when there was mild memory pressure, it would go through the page queue(s) and free clean disk pages and skip other page; only taking action on a second pass if the first pass didn't clean enough. This usually meant your program pages would stay in memory, but when you hit memory pressure, there would be a big pause to mark a lot of pages inactive, often too many pages, which would then get faulted back to active... Current FreeBSD marks pages inactive on a more consistent basis, which is nice because when there is memory pressure, chancws are there's already classified pages. But it can lead to anonymous pages getting swapped out in favor of disk pages as you described; it's all tunable, of course, but it was a kind of weird transition for me. After upgrading the OS, some of my heavy i/o machines saw rising swap usage running the same software as before; took a while to figure that out.

tonyarkles 3y ago
Ran into an issue like this on relatively modern embedded Linux platform. It was a driver bug and it’s been fixed, but here’s the scenario: - heavy disk access because we were writing real-time images to an SSD at about 500MB/s - our application was steady-state about 4GB of RAM and we had 32GB available on the platform - the serial port that we received data from was, under the hood, using DMA In certain cases, Linux would completely run out of free pages (28GB of it being used for cache on files we were never going to read again). These were all available pages but just occupied at the exact moment. The serial driver would request a page for DMA when it received an interrupt and being inside an interrupt context would request that page with NOBLOCK. That meant that kmalloc would return NULL instead of giving a page, since it would need to evict one of the cache pages before one was available. The serial driver would then blow up and never retry the DMA transaction. Fun to debug that one!
- toast0 3y ago
  DMA pages are fun. Some devices have special needs for DMA buffers, so maybe you've got something ancient that can only use memory under 32-bit, or maybe you have something really ancient that can only use memory under 16 MB; or maybe the disk controller is fine for regular disk access, but administrative commands need to use a limited range. I didn't really finish debugging that one, I got close enough and said well --- we can just measure SMART status a lot less frequently on boxes with that controller and called it a day. :)