10 ms·

> On Linux write to a regular file never blocks. Writing to a file copies data from our user space buffer to the kernel buffer and returns immediately. At some

> On Linux write to a regular file never blocks. Writing to a file copies data from our user space buffer to the kernel buffer and returns immediately. At some later point in time the kernel will send it to the disk. A regular file is hence always ready for writing and epoll wouldn’t add anything. Is that true? If it would be, the amount of data the Kernel would need to buffer is unbounded. I assumed there is a limit on the amount of buffered and not yet committed data, and when that is cross the call would block until more data is flushed to disk. Which is kind of the same as happens for TCP sockets. The `write()` call there doesn't really send data to the peer, it just submits data to the kernels send buffer, from where it will be asynchronously transmitted. Edit: Actually I will answer my own question and say I know it will block. I had deployed IO heavy applications in the past with instrumented read/write calls for IO operations in a threadpool. Even though typical IO times are well below 1ms, under extremely high load latencies of more than 1s could be observed, which is far from "not blocking".

kotlin2 4y ago
The write call returns how many bytes were accepted: https://man7.org/linux/man-pages/man2/write.2.html https://man7.org/linux/man-pages/man2/write.2.html > The number of bytes written may be less than count if, for example, there is insufficient space on the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)), or the call was interrupted by a signal handler after having written less than count bytes. (See also pipe(7).)
- wtallis 4y ago
  That doesn't answer the question. Blocking isn't a matter of how much data is written, but a matter of when the system call completes. Other parts of that man page imply that write(2) may block, unless the fd was opened with O_NONBLOCK (in which case you'll get an EAGAIN error instead of it blocking).
  - throwaway09223 4y ago
    > (in which case you'll get an EAGAIN error instead of it blocking). You won't. O_NONBLOCK cannot be used with regular files. That part of the manpage is discussing other non-socket file types. Disk i/o via write(2) is always a blocking call. Always. 100% of the time, no exceptions.
  - icedchai 4y ago
    "It's complicated." Generally, with a regular file, write(2) will complete as soon as the the data makes it to filesystem buffers/cache. The data is probably not on disk when the call completes. This depends on how the file was opened (O_FSYNC, O_DIRECT, etc.) and the underlying filesystem itself. There are many other details at work, like actual file system, memory pressure (there may not be enough buffers), cache in the physical disk device or controller, etc. So the write call itself is "blocking", but the physical writes are (generally) not synchronous with the call.
    - wtallis 4y ago
      Yes, whether a write blocks is really about whether the application can do anything else while the write is processed; whether the application is told the write is done when it lands in a cache or when it is actually on stable storage is a separate question.
Snild 4y ago
> I assumed there is a limit on the amount of buffered and not yet committed data, and when that is cross the call would block until more data is flushed to disk. There is. It's tunable through /proc/sys/vm/dirty_ratio. When there is that much write cache, application writes will start to writeback synchronously. There is also dirty_background_ratio, which is the threshold at which writeback starts happening in the background (that is, in a kernel thread).
throwaway09223 4y ago
No, as you reasoned out it is absolutely incorrect. Write calls to regular files will block until they are complete, unless some kind of error situation is encountered. This effect is often particularly pronounced with NFS, where calls might block for hours or even indefinitely if the underlying network filesystem goes away.
- inetknght 4y ago
  > This effect is often particularly pronounced with NFS, where calls might block for hours or even indefinitely if the underlying network filesystem goes away. I can't tell you how many times I've had to debug a stuck process and it turns out that the logs indicated the NFS had a hiccup a day or two ago during a file read or write and the process was never notified of a file error. It's f!@#ing frustrating. Worse, though, was CIFS.
  - lanstin 4y ago
    I routinely have to run file system scans on a giant NFS filer, and even without a hiccup, out of a 100M stat or read calls, ten or so will just never finish. In Go, I have to wrap the call with a channel thing and a time out and hope I don’t run out of threads before scanning all 400 M files.
- tankenmate 4y ago
  Just in case anyone isn't aware there is a mount flag called "soft" that allows the NFS client (and some other network filesystems) to timeout or be interrupted, i.e. the process won't get stuck in 'D' (device wait) state.
- remram 4y ago
  That is the other extreme, and is also wrong. write(2) does not block until the write to disk is completed, that's what we have fsync(2).
  - throwaway09223 4y ago
    No, that is incorrect. I think you're confused as to what "blocking syscall" means. A blocking syscall doesn't always block, but it can always block because its interface is designed to block. Write(2) to a disk file can always block, 100% of the time. It is always considered a blocking syscall, even when it happens to not take very long due to cache optimizations. Every single blocking syscall can sometimes return right away. This has no bearing on whether it is, or isn't, a potentially blocking call.
kentonv 4y ago
Yes, file I/O can block. However, there is an assumption that file I/O will never block "indefinitely" -- unless something is severely broken, the kernel will always finish the operation in finite time probably measured in milliseconds at most. The same is not true of network communications, where you may be waiting for an event that never happens. There is a temptation to say that, well, milliseconds are a long time, so wouldn't we like to do this in a non-blocking way so we can work on other stuff in the meantime? But... consider this: Reads and writes of memory also may block. If you really think about it, the only real difference between main memory blocking and disk blocking is the amount of time they may block. And with modern SSDs that time difference is not as large as it used to be. So do you want to be able to access memory in a non-blocking way? Well... you can make the same logical arguments as you do with file I/O, but in practice, almost no one tries to do this. Instead, you separate work into threads, and let the CPU switch (hyper)threads whenever it needs to wait for memory. In fact, memory reads may very well block on disk, if you use swap! Given all this, it stops being so clear that async file I/O really makes sense. Meanwhile, as it happens, the Linux kernel was never really designed for async file I/O in the first place. When you perform file I/O, the kernel may need to execute filesystem driver code, and it does so within the same thread that invoked the operation from userspace. That filesystem code is blocking. For the kernel to deliver true async file I/O, either all this code needs to be rewritten to be non-blocking (which would probably slow it down in most cases!), or the kernel needs to start a thread behind the scenes to perform the work. But... you can just as easily start a thread in userspace. So... maybe just do that? (Or, the modern answer: Use io_uring, which is explicitly designed to allow a userspace thread to request work performed on a separate kernel thread, and get notified of completion later.)
- loeg 4y ago
  > For the kernel to deliver true async file I/O, either all this code needs to be rewritten to be non-blocking This is, I believe, the NT model.
- deleted 4y ago
  [deleted]
- abiloe 4y ago
cout 4y ago
It is bounded by available memory. Writes to a socket go to a FIFO queue (the socket's write buffer), but writes to disk are different; they go through the page cache (https://www.kernel.org/doc/html/latest/admin-guide/mm/concepts.html https://www.kernel.org/doc/html/latest/admin-guide/mm/concep...): > The physical memory is volatile and the common case for getting data into the memory is to read it from files. Whenever a file is read, the data is put into the page cache to avoid expensive disk access on the subsequent reads. Similarly, when one writes to a file, the data is placed in the page cache and eventually gets into the backing storage device. The written pages are marked as dirty and when Linux decides to reuse them for other purposes, it makes sure to synchronize the file contents on the device with the updated data. There are many advantages to doing it this way. One is that multiple writes to the same page will result in a single physical write, if the page has not yet been flushed to disk. There are many reasons that you might have seen a write to a file block. One is that the number of dirty pages has reached the threshold (nr_dirty_threshold in /proc/vmstat). After that happens, any process doing disk IO will block. Another reason is memory pressure. Since all writes go through the page cache, the kernel must first allocate a page before the call to write(2) can be completed. If there are many pages in the page cache, this can take a long time (I once witnessed an old kernel bug cause all page allocations to result in kswapd attempting to reclaim pages, due to active pages being placed ahead of inactive pages in the LRU lists). In general, if you are writing a lot to disk but you have no intention of reading it in the near future, it is a good idea to call posix_fadvise(2) with FADV_DONTNEED to ensure the pages will be reused for something else more quickly.
- lanstin 4y ago
  It is pretty easy to completely hork a large box with a very disk intensive process; hit a local file system hard enough and you can get a majority of the processes into D state, uninterruptible IO Disk wait. Maybe not from inside a container, haven’t see it, but definitely on a box with shared processes. Even just too much logging can harm unrelated processes that aren’t even doing much with the disk.
caf 4y ago
The term "blocking" in UNIX-like OSes is jargon with a particular meaning. It means an interruptible wait. Disk files do not block - they may Disk Wait instead, which is an uninterruptible wait (this is what the 'D' process status stands for). Disk Wait doesn't interact with O_NONBLOCK, select(3), poll(3) etc. (Back in the bad old days it wasn't even possible for a Disk Waiting process to wake up to process a SIGKILL and die, which was the bane of system administrators everywhere when NFS introduced the idea of disks that could disappear when the network went down. Now it's common for OSes to make some kinds of Disk Waits at least killable).
- pram 4y ago
  Yep, last time I janitored NFS at scale anything stuck in D required an entire system reboot.
fsckboy 4y ago
I don't know if I'm contributing anything, but to clarify by going back to basics (rather than advanced or non standard system calls), all writes (and reads) are individually atomic, so if you and I are writing to the same file at the same time (in real time), how our overwriting behaves is relatively well behaved and can be relied on, so there is at least that level of atomic blocking, and that's important because the kernel also buffers files transparently, in that our two processes will share the same buffers, so beyond atomicity in a shared piece of the file, we won't get blocked wrt each other. But your reads need to pick up my writes (atomicity!) so there isn't going to be buffered queuing of writes on top of the disk buffering. so if the kernel decides it needs to spend more time syncing dirty buffers to disk, it's not blocking our processes in the regard above, but the system could become net i/o bound. This is 1/2 the reason to make sure to use virtual memory, so plenty of a flexible amount of disk buffer will be available (the other half being to swap out resident but no longer referenced dirty pages, as from the initialization code in your app)
graham_king_3 4y ago
Thanks for starting a very interesting thread. I re-wrote part of that section based on all the feedback.

kotlin2 4y ago

The write call returns how many bytes were accepted: https://man7.org/linux/man-pages/man2/write.2.html https://man7.org/linux/man-pages/man2/write.2.html > The number of bytes written may be less than count if, for example, there is insufficient space on the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)), or the call was interrupted by a signal handler after having written less than count bytes. (See also pipe(7).)

wtallis 4y ago
That doesn't answer the question. Blocking isn't a matter of how much data is written, but a matter of when the system call completes. Other parts of that man page imply that write(2) may block, unless the fd was opened with O_NONBLOCK (in which case you'll get an EAGAIN error instead of it blocking).
- throwaway09223 4y ago
  > (in which case you'll get an EAGAIN error instead of it blocking). You won't. O_NONBLOCK cannot be used with regular files. That part of the manpage is discussing other non-socket file types. Disk i/o via write(2) is always a blocking call. Always. 100% of the time, no exceptions.
- icedchai 4y ago
  "It's complicated." Generally, with a regular file, write(2) will complete as soon as the the data makes it to filesystem buffers/cache. The data is probably not on disk when the call completes. This depends on how the file was opened (O_FSYNC, O_DIRECT, etc.) and the underlying filesystem itself. There are many other details at work, like actual file system, memory pressure (there may not be enough buffers), cache in the physical disk device or controller, etc. So the write call itself is "blocking", but the physical writes are (generally) not synchronous with the call.
  - wtallis 4y ago
    Yes, whether a write blocks is really about whether the application can do anything else while the write is processed; whether the application is told the write is done when it lands in a cache or when it is actually on stable storage is a separate question.

Snild 4y ago

> I assumed there is a limit on the amount of buffered and not yet committed data, and when that is cross the call would block until more data is flushed to disk. There is. It's tunable through /proc/sys/vm/dirty_ratio. When there is that much write cache, application writes will start to writeback synchronously. There is also dirty_background_ratio, which is the threshold at which writeback starts happening in the background (that is, in a kernel thread).

throwaway09223 4y ago

No, as you reasoned out it is absolutely incorrect. Write calls to regular files will block until they are complete, unless some kind of error situation is encountered. This effect is often particularly pronounced with NFS, where calls might block for hours or even indefinitely if the underlying network filesystem goes away.

inetknght 4y ago
> This effect is often particularly pronounced with NFS, where calls might block for hours or even indefinitely if the underlying network filesystem goes away. I can't tell you how many times I've had to debug a stuck process and it turns out that the logs indicated the NFS had a hiccup a day or two ago during a file read or write and the process was never notified of a file error. It's f!@#ing frustrating. Worse, though, was CIFS.
- lanstin 4y ago
  I routinely have to run file system scans on a giant NFS filer, and even without a hiccup, out of a 100M stat or read calls, ten or so will just never finish. In Go, I have to wrap the call with a channel thing and a time out and hope I don’t run out of threads before scanning all 400 M files.
tankenmate 4y ago
Just in case anyone isn't aware there is a mount flag called "soft" that allows the NFS client (and some other network filesystems) to timeout or be interrupted, i.e. the process won't get stuck in 'D' (device wait) state.
remram 4y ago
That is the other extreme, and is also wrong. write(2) does not block until the write to disk is completed, that's what we have fsync(2).
- throwaway09223 4y ago
  No, that is incorrect. I think you're confused as to what "blocking syscall" means. A blocking syscall doesn't always block, but it can always block because its interface is designed to block. Write(2) to a disk file can always block, 100% of the time. It is always considered a blocking syscall, even when it happens to not take very long due to cache optimizations. Every single blocking syscall can sometimes return right away. This has no bearing on whether it is, or isn't, a potentially blocking call.

kentonv 4y ago

Yes, file I/O can block. However, there is an assumption that file I/O will never block "indefinitely" -- unless something is severely broken, the kernel will always finish the operation in finite time probably measured in milliseconds at most. The same is not true of network communications, where you may be waiting for an event that never happens. There is a temptation to say that, well, milliseconds are a long time, so wouldn't we like to do this in a non-blocking way so we can work on other stuff in the meantime? But... consider this: Reads and writes of memory also may block. If you really think about it, the only real difference between main memory blocking and disk blocking is the amount of time they may block. And with modern SSDs that time difference is not as large as it used to be. So do you want to be able to access memory in a non-blocking way? Well... you can make the same logical arguments as you do with file I/O, but in practice, almost no one tries to do this. Instead, you separate work into threads, and let the CPU switch (hyper)threads whenever it needs to wait for memory. In fact, memory reads may very well block on disk, if you use swap! Given all this, it stops being so clear that async file I/O really makes sense. Meanwhile, as it happens, the Linux kernel was never really designed for async file I/O in the first place. When you perform file I/O, the kernel may need to execute filesystem driver code, and it does so within the same thread that invoked the operation from userspace. That filesystem code is blocking. For the kernel to deliver true async file I/O, either all this code needs to be rewritten to be non-blocking (which would probably slow it down in most cases!), or the kernel needs to start a thread behind the scenes to perform the work. But... you can just as easily start a thread in userspace. So... maybe just do that? (Or, the modern answer: Use io_uring, which is explicitly designed to allow a userspace thread to request work performed on a separate kernel thread, and get notified of completion later.)

loeg 4y ago
> For the kernel to deliver true async file I/O, either all this code needs to be rewritten to be non-blocking This is, I believe, the NT model.
deleted 4y ago
[deleted]
abiloe 4y ago

cout 4y ago

It is bounded by available memory. Writes to a socket go to a FIFO queue (the socket's write buffer), but writes to disk are different; they go through the page cache (https://www.kernel.org/doc/html/latest/admin-guide/mm/concepts.html https://www.kernel.org/doc/html/latest/admin-guide/mm/concep...): > The physical memory is volatile and the common case for getting data into the memory is to read it from files. Whenever a file is read, the data is put into the page cache to avoid expensive disk access on the subsequent reads. Similarly, when one writes to a file, the data is placed in the page cache and eventually gets into the backing storage device. The written pages are marked as dirty and when Linux decides to reuse them for other purposes, it makes sure to synchronize the file contents on the device with the updated data. There are many advantages to doing it this way. One is that multiple writes to the same page will result in a single physical write, if the page has not yet been flushed to disk. There are many reasons that you might have seen a write to a file block. One is that the number of dirty pages has reached the threshold (nr_dirty_threshold in /proc/vmstat). After that happens, any process doing disk IO will block. Another reason is memory pressure. Since all writes go through the page cache, the kernel must first allocate a page before the call to write(2) can be completed. If there are many pages in the page cache, this can take a long time (I once witnessed an old kernel bug cause all page allocations to result in kswapd attempting to reclaim pages, due to active pages being placed ahead of inactive pages in the LRU lists). In general, if you are writing a lot to disk but you have no intention of reading it in the near future, it is a good idea to call posix_fadvise(2) with FADV_DONTNEED to ensure the pages will be reused for something else more quickly.

lanstin 4y ago
It is pretty easy to completely hork a large box with a very disk intensive process; hit a local file system hard enough and you can get a majority of the processes into D state, uninterruptible IO Disk wait. Maybe not from inside a container, haven’t see it, but definitely on a box with shared processes. Even just too much logging can harm unrelated processes that aren’t even doing much with the disk.

caf 4y ago

The term "blocking" in UNIX-like OSes is jargon with a particular meaning. It means an interruptible wait. Disk files do not block - they may Disk Wait instead, which is an uninterruptible wait (this is what the 'D' process status stands for). Disk Wait doesn't interact with O_NONBLOCK, select(3), poll(3) etc. (Back in the bad old days it wasn't even possible for a Disk Waiting process to wake up to process a SIGKILL and die, which was the bane of system administrators everywhere when NFS introduced the idea of disks that could disappear when the network went down. Now it's common for OSes to make some kinds of Disk Waits at least killable).

pram 4y ago
Yep, last time I janitored NFS at scale anything stuck in D required an entire system reboot.

fsckboy 4y ago

I don't know if I'm contributing anything, but to clarify by going back to basics (rather than advanced or non standard system calls), all writes (and reads) are individually atomic, so if you and I are writing to the same file at the same time (in real time), how our overwriting behaves is relatively well behaved and can be relied on, so there is at least that level of atomic blocking, and that's important because the kernel also buffers files transparently, in that our two processes will share the same buffers, so beyond atomicity in a shared piece of the file, we won't get blocked wrt each other. But your reads need to pick up my writes (atomicity!) so there isn't going to be buffered queuing of writes on top of the disk buffering. so if the kernel decides it needs to spend more time syncing dirty buffers to disk, it's not blocking our processes in the regard above, but the system could become net i/o bound. This is 1/2 the reason to make sure to use virtual memory, so plenty of a flexible amount of disk buffer will be available (the other half being to swap out resident but no longer referenced dirty pages, as from the initialization code in your app)

graham_king_3 4y ago

Thanks for starting a very interesting thread. I re-wrote part of that section based on all the feedback.