3 ms·
Same goes for I/O scheduling. The virtual memory subsystem seems to only have two modes: buffering and write-out. The system will buffer writes until some thres
by tremon 3y ago
Same goes for I/O scheduling. The virtual memory subsystem seems to only have two modes: buffering and write-out. The system will buffer writes until some threshold is reached, then it switches to write-out mode and flushes all dirty pages regardless of their age. The problem is that new pages are locked during write-out, so pending writes will stall, regardless of the memory pressure on the rest of the system.
A better design would be to only lock a subset of dirty pages and let new writes to virtual memory continue while the write-out is happening, but it doesn't seem like the system can accomodate that.
A simple test to see this in action is to use netcat to transfer a large file across the network, and monitor the device activity with sar or atop (or just check the disk/network activity lights). What you'll see is that while the disk is writing, network activity drops to zero, and when network activity resumes, the disk remains idle again for seconds. It doesn't matter how much smaller you make vm.dirty_background_{bytes,ratio} with respect to vm.dirty_{bytes,ratio}, the network traffic will block as soon as the "background" disk write starts. The only effect a low value for vm.dirty_background_{bytes,ratio} has is to increase the frequency at which the network and disk activity will alternate.
- freedomben 3y agoWhy do you think it's still this way? Why hasn't somebody implemented something better? Is it too core that we can't afford any instability? Edit: apparently it has been done, but only very recently and hasn't had a major distro ship it yet
- ahartmetz 3y agoOh? That sounds important and interesting. Do you have a URL or commit hash with more information?
- freedomben 3y agoI'm very confused now so I have no idea what to think. I need to take some time to read through everything, but in the mean time here are some relevant links I've seen: https://lwn.net/Articles/925371/ https://lwn.net/Articles/925371/ https://lkml.org/lkml/2016/4/23/135 https://lkml.org/lkml/2016/4/23/135
- ahartmetz 3y agoHm yes, maybe you are confused. The articles you posted seem to be about CPU, not I/O.
- alyandon 3y agoRan into similar pathological behavior on servers with huge amounts of ram (>1 TiB). Negative dentries builds up until the kernel decides it is time to reclaim and locks up the server for >60 seconds scanning pages. :-/
- tremon 3y agoYes, indeed. The more ram available, the more pages are available as buffer cache, the more pronounced the system stalls are. Though in my example, the determining factor is available_memory/disk_write_troughput, while in your example it is available_memory/single_core_throughput (assuming the page scan is a single-thread process).
- mkhnews 3y agoDoes disabling transparent huge pages help any here ?
- tremon 3y agoHave not tried it, but I would not expect any difference. FAFAIK, transparent huge pages are only about organizing free/unused/allocated memory; in both mine and the parent's example, the issue isn't about memory management per se but about synchronous operations on data taking too long (I/O write in my case, cache cleanup for the GP).
- anonacct37 3y agoI also ran into spooky negative dentry issues. On a fleet of seemingly identical physical machines one machine would constantly check for files that didn't exist (iirc it was somehow part of nss and a quick Google search seems to confirm https://lwn.net/Articles/894098/ https://lwn.net/Articles/894098/ ). I'm okayish at systems level debugging but I never figured that one out. It caused the kernel to use alot of memory. Arguable whether or not it impacted performance since it was a cache.
- alexey-salmin 3y agoA complete nightmare. In the past I had to work on an IO-heavy system with 1-5ms read-latency guarantees and the only working solution was to completely manage all writes in the userspace. Manual buffering, manual throttling, manual fallocate, then either DIRECT_IO or mmap+sync_file_range depending on whether you need the written data to be readily available for reads or not. There was a kernel patch though that solved around 95% of issues with no userspace changes: vm.dirty_write_behind. Unfortunately it never made it into the mainline kernel. [1] For strong latency guarantees it ws insufficient but it greatly improved the typical network/IO spike alternations described above. I'm surprised it was never fixed upstream despite that fact that even most basic and simple scenarios like "nginx writes a log file to the disk" sometimes explode with seconds-long lockups on memory-fat machines. [1] https://lore.kernel.org/linux-fsdevel/156896493723.4334.13340481207144634918.stgit@buzz/ https://lore.kernel.org/linux-fsdevel/156896493723.4334.1334...
- tux1968 3y ago> Unfortunately it never made it into the mainline kernel. That is unfortunate, and a bit surprising given how positively both Linus Torvalds, and Jens Axboe, responded to the patch.
- viraptor 3y agoWould a solution to this include some kind of new queue? It is the fsync barrier enough for synchronising the writes? I'm thinking of situations where you try to write a subset of blocks, but you end up writing some new ones before old ones.
- tremon 3y agoI'm not sure how the current vm system tracks dirty pages, so I wouldn't know where to start suggesting realistic solutions. But yes, assuming the dirty pages are organized in some sort of specialized data structure (queue or otherwise), you could do this without needing new barriers: it would suffice to create a new dirty-page root structure before starting writeout, and redirect freshly dirtied pages to the new root. The writer process can then take its leisurely time committing the pending writes to disk without obstructing other processes issuing new writes. New writes to pending pages can be reparented to the new root via an RCU mechanism (to not violate previous write barriers), and incomplete writes can be reparented by the writer process in a similar way. When the writeout process completes, the old root can be completely discarded; the writer could even consume and free the old structure while traversing it. But really, this is all fantasy given that I know nothing about the virtual memory internal data structures.
- afr0ck 3y agoThe kernel page cache is an XArray [1] (data structure here [2]) and dirty page pointers are annotated with the dirty tag when the page is dirtied [3]. There is a per-filesystem callback that usually always ends up here [4], where you can see the the above data structure [2] entry for that page is being tagged as dirty. Edit: fixed typo [1] https://docs.kernel.org/core-api/xarray.html https://docs.kernel.org/core-api/xarray.html [2] https://elixir.bootlin.com/linux/latest/source/include/linux/fs.h#L472 https://elixir.bootlin.com/linux/latest/source/include/linux... [3] https://elixir.bootlin.com/linux/latest/source/mm/page-writeback.c#L2763 https://elixir.bootlin.com/linux/latest/source/mm/page-write... [4] https://elixir.bootlin.com/linux/latest/source/mm/page-writeback.c#L2670 https://elixir.bootlin.com/linux/latest/source/mm/page-write...