7 ms·

Disk write buffering and its interactions with write flushes

76 points by ingve 3y ago

mgerdts 3y ago
> Rather than allowing multiple gigabytes of outstanding buffered writes and deferring writeback until a gigabyte or more has accumulated, you'd set things to trigger writebacks almost immediately and then force processes doing write IO to wait for disk writes to complete once you have more than a relatively small volume of outstanding writes. This is especially true when the thing doing a bunch of buffered writes is in a VM. If the VMM is buffering writes to the host fs, you get the described effects in the host OS and the guest OS.
- Borg3 3y ago
  And thats what direct I/O is for. Leave that to the guest OS.
  - mgerdts 3y ago
    Edit: maybe you were suggesting the host does direct IO. That is exactly what I recommend. I initially read this differently. Original: If you are running the host OS you often have little or no say about what happens in the guest OS. Even if you control both it is likely not trivial to get the apps to use direct IO. It could even be harmful to use direct IO because that would mean that writes would not stay in the guest buffer cache, forcing what would have been a cached read or minor fault in the guest into what it sees as a physical read. The written blocks are not going to be shared, except maybe due to KSM. But KSM would do the same if that data was in the guest’s buffer cache if huge pages are not used.
    - Borg3 3y ago
      Of course I was talking about host doing direct I/O :) Glad you got it right.
M95D 3y ago
I noticed this problem when I switched to Linux. On Windows, when I copied a file, disk writes started immediately. On older systems, like Win98, I had to tweak Total Commander's disk buffer to improve copy speed on the same drive. Total Commander even had separate settings for same disk vs. different disk copy buffer sizes. When I switched to Linux I was immediately surprised that disk writes did not start until the memory was full, and then it would stop reading while flushing dirty data. This happens even if the copy is between different drives: reads stop, writes only, then reads again with no writes to the other disk, repeat. It basically halves the copy speed. It even happens when I copy to network mounts: reads 20 GB of data into memory, then reading stops and tries to flush the data over the nfs. Nfs times out, transfer fails. I had to use nfs timouts of 1h just to be able to do a backup. It drives me crazy. Is there any way to make it write immediately, or at least to put a memory limit on dirty data?
- nolist_policy 3y ago
  https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-bytes https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-byt... https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-background-bytes https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-bac... The values are crazy high by default (on modern hardware anyway): 10% of memory for dirty_background_bytes and 20% for dirty_bytes. I wonder why no distro touches these.
  - suprjami 3y ago
    Because people complain their system is "slow" if it blocks on disk I/O. Another set of people also complain Linux takes too long to safely unplug USB drives.
    - Rygian 3y ago
      When the choice is between "slow" file transfers and being unable to do file transfers because of NFS timeouts, the choice should be obvious. I've lost data as a side effect of a simple file transfer timing out.
pradn 3y ago
fsync() guarantees that writes have hit the disk. But is there a guarantee about what's written before an fsync()? Can it be anywhere between "nothing" and "everything"? I suppose this must be a loose guarantee if the "write-back" parameter can be tweaked at will.
- nolist_policy 3y ago
  > Can it be anywhere between "nothing" and "everything"? Yes, and that won't change because the hardware with it's own buffers behaves the same way.
- toast0 3y ago
  > Can it be anywhere between "nothing" and "everything"? Yes, and there's not (generally) any ordering constraint, either. The last thing you wrote may be persisted, and not the first, etc.
- loeg 3y ago
  There's some user input to this via posix_fadvise POSIX_FADV_DONTNEED but it doesn't guarantee anything.
- vlovich123 3y ago
  If you're generating an immutable file, a common technique is to play fsync + rename tricks although I think a more modern technique would be: 1. open directory (dir_fd) 2. create an unnamed O_TMPFILE file (file_fd) 3. write to file_fd 4. fdatasync(file_fd) 5. linkat(dir_fd, "", dir_fd, "file name", AT_EMPTY_PATH) 5. fsync(dir_fd) This should guarantee that either "file name" will have the old contents or the new contents and no transient version is observed. The "old" mechanism is similar in that you write out to a temporary sibling and rename (these days you'd use RENAME_EXCHANGE w/ renameat2 to guarantee the atomicity or get an error) with the differentiating difference being that the temporary data could be observed on the filesystem / left around if you have a machine reboot.
  - vlovich123 3y ago
    Correction to typo: 5. linkat(file_fd, "", dir_fd, "file name", AT_EMPTY_PATH)
Rygian 3y ago
My systems routinely report transferring files at speeds much faster than what the physical medium asked, only to have them get stuck and fail on a time out a while after. Having such behavior be the default is, to my limited understanding, a bug in the Linux kernel.
- adr1an 3y ago
  There's "$ sync" to ask the Kernel to start the actual write. And "dd" command has the option to do so, too. It's just not the default, as many things on Linux, unfortunately.
- JackSlateur 3y ago
  You can see this the other way When I write lots of data to a rusty device and don't sync, the kernel gives back more quickly. From there, I can move on with my work : everything is "as-if" the data were actually copied (except power-cutoff protection, indeed) As such, I see this behavior as a feature, it allows me to wait less and do my work quicker. Unless windows, on which I must wait for the whole data to be written on the disk.
  - Rygian 3y ago
    > As such, I see this behavior as a feature, it allows me to wait less and do my work quicker. So, when (if?) you notice that the transfer has failed with a timeout, do you still consider your work was done quicker?
    - JackSlateur 3y ago
      There is no timeout possible for IO cache If you are thinking of something like a HTTP download, the transfer has to be done anyway (and I must wait for it). However, I do not need that data to be fully written to the (potentially busy) local device
      - Rygian 3y ago
        My initial comment was about a simple 'cp' copy of a large file from local storage to an NFS mount. It times out, 'cp' exits in error, and my file is not transferred. I can reproduce this at will. I think I should report it as a linux kernel bug, to be honest.
toast0 3y ago
> Rather than allowing multiple gigabytes of outstanding buffered writes and deferring writeback until a gigabyte or more has accumulated, you'd set things to trigger writebacks almost immediately and then force processes doing write IO to wait for disk writes to complete once you have more than a relatively small volume of outstanding writes. I think having the trigger be size based rather than timebased is the real problem. Or bounds on both... I probably don't want to buffer writes for more than X seconds, or let the buffer grow beyond Y% of ram. At least for the time based limit, you'd really want to be able to say start writing to disk when the buffer has data over 10 seconds old, but still accept writes into a new buffer, only blocking writes when there's a buffer being written out and the current buffer is too old or too big.
- loeg 3y ago
  Ideally the system starts flushing buffered writes basically immediately, but with low device-level queue depth so that subsequently issued higher priority IOs do not suffer from a ton of additional latency.
  - kvemkon 3y ago
    Ideally is only when the size of the complete file is known in advance. To minimize avoidable fragmentation.
    - loeg 3y ago
      Eh. At the rates we're talking about (keeping the disk 100% busy for seconds at a time) you're writing big enough chunks that isn't an issue.
- magicalhippo 3y ago
  ZFS has both a time-based and size-based limit. IIRC the default is 15 seconds and lower of 2GB and some % of system memory. Though in ZFS' case it's not really a regular write cache as such, as it's used to minimize updates to its on-disk copy-on-write structure.
- 3y ago
randkyp 3y ago
I found this recent thread interesting, specifically about really considering whether you're going to read the data you just wrote in the near future or not (in which case, use direct IO) and a set of (abandoned?) patches for write-behind caching for sequential writes in Linux (https://lore.kernel.org/lkml/156896493723.4334.13340481207144634918.stgit@buzz/ https://lore.kernel.org/lkml/156896493723.4334.1334048120714...). https://lore.kernel.org/linux-mm/45odvhgymm7fxsgwpewoiiggaokxjcr7ootsamm4rwsdw26j62@5lohhv2kvofo/T/ https://lore.kernel.org/linux-mm/45odvhgymm7fxsgwpewoiiggaok...
- rwmj 3y ago
  Direct IO is very inflexible. A better way is this recommended by Linus in an old post: https://lkml.iu.edu/hypermail/linux/kernel/1005.2/01845.html https://lkml.iu.edu/hypermail/linux/kernel/1005.2/01845.html I implemented something along the same lines but a bit less spicy here: https://gitlab.com/nbdkit/nbdkit/-/commit/aa5a2183a6d16afd919174b5ba8222b2bccf4039 https://gitlab.com/nbdkit/nbdkit/-/commit/aa5a2183a6d16afd91...
  - rwmj 3y ago
    That second link is wrong, should be: https://gitlab.com/nbdkit/nbdkit/-/commit/a956e2e75d6c88eeefecd967505667c9f176e3af https://gitlab.com/nbdkit/nbdkit/-/commit/a956e2e75d6c88eeef...
hsaliak 3y ago
The theory is compelling but this is the exact sort of problem that needs a representative benchmark. Based on the benchmark, the optimal strategy can then be picked. I don’t see any stats/graphs or benchmarks in the linked article.
cafxx 3y ago
I think that the idea of not initiating writeback immediately derives mostly from the days of spinning rust, where read latencies would be noticeably impacted if you initiated writeback too aggressively: reads, contrary to writes, are synchronous by default, and spinning rust rarely allowed high (by modern standards) IOPS, so it made a lot of sense to buffer writes as much as possible to minimize the number of I/O operations spent on writes, as this would leave as many of those IOPS as possible available for reads. This is probably much less of a concern today, as NVMe drives - beside having many orders of magnitude higher IOPS capacity - also have (at least on paper) much better hardware support for high I/O concurrency. It may still make sense, even today, if your hardware (or stack) limits IOPS.
- vlovich123 3y ago
  As I mention elsewhere in the thread, reading the kernel documentation for the various flags suggests that the kernel devs are also concerned about multiple writes to the same piece of data & thus buffering lets you potential elide unnecessary disk I/O.
pengaru 3y ago
One of the things I improved in my recordMyDesktop fork [0] was an awful tendency for the frame cache writer to accumulate heaps of dirty pages until background writeback would flush them out. I had 16GiB of RAM which meant quite large swaths of dirty pages would become buffered while the SSD sat idle until writeback began. This would cause high-FPS full-screen recordings in particular to become backlogged and start dropping frames / audio dropouts. Just generally broken behavior for a desktop recorder, especially for a deferred-encode mode that's supposed to be optimized for minimizing system-wide effects/overheads during the recording. The simple solution I found was to proactively initiate writeback regularly via fdatasync() on the cache fd. [1] I haven't decided yet if more should be done to constrain its buffer cache effects though. The cache files will be read back during encoding in post, so if there's enough RAM it can be desirable to enable reading them back entirely from memory instead of having to hit the disk again... but it would also be nice to let the rest of the system's processes keep their stuff in the page cache. memcg can probably be used to find a balanced solution, but I haven't done any experiments yet. Have any of you handled similar scenarios? What did you do? [0] https://github.com/recordmydesktop/recordmydesktop https://github.com/recordmydesktop/recordmydesktop [1] https://github.com/recordmydesktop/recordmydesktop/commit/42904eef310313eeed6f8d91d5306488fd016d2e https://github.com/recordmydesktop/recordmydesktop/commit/42...

76 points by ingve 3y ago

Borg3 3y ago
And thats what direct I/O is for. Leave that to the guest OS.
- mgerdts 3y ago
  Edit: maybe you were suggesting the host does direct IO. That is exactly what I recommend. I initially read this differently. Original: If you are running the host OS you often have little or no say about what happens in the guest OS. Even if you control both it is likely not trivial to get the apps to use direct IO. It could even be harmful to use direct IO because that would mean that writes would not stay in the guest buffer cache, forcing what would have been a cached read or minor fault in the guest into what it sees as a physical read. The written blocks are not going to be shared, except maybe due to KSM. But KSM would do the same if that data was in the guest’s buffer cache if huge pages are not used.
  - Borg3 3y ago
    Of course I was talking about host doing direct I/O :) Glad you got it right.

M95D 3y ago

I noticed this problem when I switched to Linux. On Windows, when I copied a file, disk writes started immediately. On older systems, like Win98, I had to tweak Total Commander's disk buffer to improve copy speed on the same drive. Total Commander even had separate settings for same disk vs. different disk copy buffer sizes. When I switched to Linux I was immediately surprised that disk writes did not start until the memory was full, and then it would stop reading while flushing dirty data. This happens even if the copy is between different drives: reads stop, writes only, then reads again with no writes to the other disk, repeat. It basically halves the copy speed. It even happens when I copy to network mounts: reads 20 GB of data into memory, then reading stops and tries to flush the data over the nfs. Nfs times out, transfer fails. I had to use nfs timouts of 1h just to be able to do a backup. It drives me crazy. Is there any way to make it write immediately, or at least to put a memory limit on dirty data?

nolist_policy 3y ago
https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-bytes https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-byt... https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-background-bytes https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-bac... The values are crazy high by default (on modern hardware anyway): 10% of memory for dirty_background_bytes and 20% for dirty_bytes. I wonder why no distro touches these.
- suprjami 3y ago
  Because people complain their system is "slow" if it blocks on disk I/O. Another set of people also complain Linux takes too long to safely unplug USB drives.
  - Rygian 3y ago
    When the choice is between "slow" file transfers and being unable to do file transfers because of NFS timeouts, the choice should be obvious. I've lost data as a side effect of a simple file transfer timing out.

pradn 3y ago

fsync() guarantees that writes have hit the disk. But is there a guarantee about what's written before an fsync()? Can it be anywhere between "nothing" and "everything"? I suppose this must be a loose guarantee if the "write-back" parameter can be tweaked at will.

nolist_policy 3y ago
> Can it be anywhere between "nothing" and "everything"? Yes, and that won't change because the hardware with it's own buffers behaves the same way.
toast0 3y ago
> Can it be anywhere between "nothing" and "everything"? Yes, and there's not (generally) any ordering constraint, either. The last thing you wrote may be persisted, and not the first, etc.
loeg 3y ago
There's some user input to this via posix_fadvise POSIX_FADV_DONTNEED but it doesn't guarantee anything.
vlovich123 3y ago
If you're generating an immutable file, a common technique is to play fsync + rename tricks although I think a more modern technique would be: 1. open directory (dir_fd) 2. create an unnamed O_TMPFILE file (file_fd) 3. write to file_fd 4. fdatasync(file_fd) 5. linkat(dir_fd, "", dir_fd, "file name", AT_EMPTY_PATH) 5. fsync(dir_fd) This should guarantee that either "file name" will have the old contents or the new contents and no transient version is observed. The "old" mechanism is similar in that you write out to a temporary sibling and rename (these days you'd use RENAME_EXCHANGE w/ renameat2 to guarantee the atomicity or get an error) with the differentiating difference being that the temporary data could be observed on the filesystem / left around if you have a machine reboot.
- vlovich123 3y ago
  Correction to typo: 5. linkat(file_fd, "", dir_fd, "file name", AT_EMPTY_PATH)

Rygian 3y ago

My systems routinely report transferring files at speeds much faster than what the physical medium asked, only to have them get stuck and fail on a time out a while after. Having such behavior be the default is, to my limited understanding, a bug in the Linux kernel.

adr1an 3y ago
There's "$ sync" to ask the Kernel to start the actual write. And "dd" command has the option to do so, too. It's just not the default, as many things on Linux, unfortunately.
JackSlateur 3y ago
You can see this the other way When I write lots of data to a rusty device and don't sync, the kernel gives back more quickly. From there, I can move on with my work : everything is "as-if" the data were actually copied (except power-cutoff protection, indeed) As such, I see this behavior as a feature, it allows me to wait less and do my work quicker. Unless windows, on which I must wait for the whole data to be written on the disk.
- Rygian 3y ago
  > As such, I see this behavior as a feature, it allows me to wait less and do my work quicker. So, when (if?) you notice that the transfer has failed with a timeout, do you still consider your work was done quicker?
  - JackSlateur 3y ago
    There is no timeout possible for IO cache If you are thinking of something like a HTTP download, the transfer has to be done anyway (and I must wait for it). However, I do not need that data to be fully written to the (potentially busy) local device
    - Rygian 3y ago
      My initial comment was about a simple 'cp' copy of a large file from local storage to an NFS mount. It times out, 'cp' exits in error, and my file is not transferred. I can reproduce this at will. I think I should report it as a linux kernel bug, to be honest.

toast0 3y ago

> Rather than allowing multiple gigabytes of outstanding buffered writes and deferring writeback until a gigabyte or more has accumulated, you'd set things to trigger writebacks almost immediately and then force processes doing write IO to wait for disk writes to complete once you have more than a relatively small volume of outstanding writes. I think having the trigger be size based rather than timebased is the real problem. Or bounds on both... I probably don't want to buffer writes for more than X seconds, or let the buffer grow beyond Y% of ram. At least for the time based limit, you'd really want to be able to say start writing to disk when the buffer has data over 10 seconds old, but still accept writes into a new buffer, only blocking writes when there's a buffer being written out and the current buffer is too old or too big.

loeg 3y ago
Ideally the system starts flushing buffered writes basically immediately, but with low device-level queue depth so that subsequently issued higher priority IOs do not suffer from a ton of additional latency.
- kvemkon 3y ago
  Ideally is only when the size of the complete file is known in advance. To minimize avoidable fragmentation.
  - loeg 3y ago
    Eh. At the rates we're talking about (keeping the disk 100% busy for seconds at a time) you're writing big enough chunks that isn't an issue.
magicalhippo 3y ago
ZFS has both a time-based and size-based limit. IIRC the default is 15 seconds and lower of 2GB and some % of system memory. Though in ZFS' case it's not really a regular write cache as such, as it's used to minimize updates to its on-disk copy-on-write structure.
3y ago

randkyp 3y ago

I found this recent thread interesting, specifically about really considering whether you're going to read the data you just wrote in the near future or not (in which case, use direct IO) and a set of (abandoned?) patches for write-behind caching for sequential writes in Linux (https://lore.kernel.org/lkml/156896493723.4334.13340481207144634918.stgit@buzz/ https://lore.kernel.org/lkml/156896493723.4334.1334048120714...). https://lore.kernel.org/linux-mm/45odvhgymm7fxsgwpewoiiggaokxjcr7ootsamm4rwsdw26j62@5lohhv2kvofo/T/ https://lore.kernel.org/linux-mm/45odvhgymm7fxsgwpewoiiggaok...

rwmj 3y ago
Direct IO is very inflexible. A better way is this recommended by Linus in an old post: https://lkml.iu.edu/hypermail/linux/kernel/1005.2/01845.html https://lkml.iu.edu/hypermail/linux/kernel/1005.2/01845.html I implemented something along the same lines but a bit less spicy here: https://gitlab.com/nbdkit/nbdkit/-/commit/aa5a2183a6d16afd919174b5ba8222b2bccf4039 https://gitlab.com/nbdkit/nbdkit/-/commit/aa5a2183a6d16afd91...
- rwmj 3y ago
  That second link is wrong, should be: https://gitlab.com/nbdkit/nbdkit/-/commit/a956e2e75d6c88eeefecd967505667c9f176e3af https://gitlab.com/nbdkit/nbdkit/-/commit/a956e2e75d6c88eeef...

hsaliak 3y ago

The theory is compelling but this is the exact sort of problem that needs a representative benchmark. Based on the benchmark, the optimal strategy can then be picked. I don’t see any stats/graphs or benchmarks in the linked article.

cafxx 3y ago

I think that the idea of not initiating writeback immediately derives mostly from the days of spinning rust, where read latencies would be noticeably impacted if you initiated writeback too aggressively: reads, contrary to writes, are synchronous by default, and spinning rust rarely allowed high (by modern standards) IOPS, so it made a lot of sense to buffer writes as much as possible to minimize the number of I/O operations spent on writes, as this would leave as many of those IOPS as possible available for reads. This is probably much less of a concern today, as NVMe drives - beside having many orders of magnitude higher IOPS capacity - also have (at least on paper) much better hardware support for high I/O concurrency. It may still make sense, even today, if your hardware (or stack) limits IOPS.

vlovich123 3y ago
As I mention elsewhere in the thread, reading the kernel documentation for the various flags suggests that the kernel devs are also concerned about multiple writes to the same piece of data & thus buffering lets you potential elide unnecessary disk I/O.

pengaru 3y ago

One of the things I improved in my recordMyDesktop fork [0] was an awful tendency for the frame cache writer to accumulate heaps of dirty pages until background writeback would flush them out. I had 16GiB of RAM which meant quite large swaths of dirty pages would become buffered while the SSD sat idle until writeback began. This would cause high-FPS full-screen recordings in particular to become backlogged and start dropping frames / audio dropouts. Just generally broken behavior for a desktop recorder, especially for a deferred-encode mode that's supposed to be optimized for minimizing system-wide effects/overheads during the recording. The simple solution I found was to proactively initiate writeback regularly via fdatasync() on the cache fd. [1] I haven't decided yet if more should be done to constrain its buffer cache effects though. The cache files will be read back during encoding in post, so if there's enough RAM it can be desirable to enable reading them back entirely from memory instead of having to hit the disk again... but it would also be nice to let the rest of the system's processes keep their stuff in the page cache. memcg can probably be used to find a balanced solution, but I haven't done any experiments yet. Have any of you handled similar scenarios? What did you do? [0] https://github.com/recordmydesktop/recordmydesktop https://github.com/recordmydesktop/recordmydesktop [1] https://github.com/recordmydesktop/recordmydesktop/commit/42904eef310313eeed6f8d91d5306488fd016d2e https://github.com/recordmydesktop/recordmydesktop/commit/42...