XFS has a history of reclaim/memory management issues. It used to have a habit of blocking on I/O when cleaning memory even with plenty of clean page cache avai

XFS has a history of reclaim/memory management issues. It used to have a habit of blocking on I/O when cleaning memory even with plenty of clean page cache available. For the longest time, I had issues on my home server (XFS on top of RAID at the time) with large latency hiccups, to the tune of several seconds. This was disrupting some real-time data ingestion and causing data loss. I thought it was about committing to disk, so I added big memory buffers, but no dice. I spent years with this annoying issue. I even had a kernel patch in to increase kernel-side buffers (which were not subject to this problem) to work around it. It wasn't even consistent. Eventually one day I got sick enough of it, and sat down trying to reproduce it. I figured out that it only happened when true memory usage (including buffer cache) was ~100%; if there was truly unused memory around, things were fine, and I could evict the buffer cache and it would fix the problem until it grew to consume all free RAM again. Eventually I managed to get a stack trace of a process that was stalling even though it wasn't anywhere near writing to disk, and I found out it was stalling during a write(). To a pipe. Because the kernel had to allocate data for the pipe buffers. And it was asking XFS. And XFS decided to evict some dirty inodes, and block on that. Even with gigabytes of clean page cache available to evict. What. Swearing ensued - here I thought I had some weird kernel/hardware issue causing latency spikes, and it was XFS all along. I eventually ripped XFS out and replaced it with ext4 and that solved the issue. This eventually got fixed in 2019: https://lwn.net/Articles/795098/ https://lwn.net/Articles/795098/