9 ms·

> In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as

by dap 13y ago

> In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. If you don't issue an fsync(), the kernel never has to write anything. If you had a different function that did what you suggest, returning when the kernel had decided to write the data (without instructing the kernel to do so soonish), you could literally be waiting forever. If the function ever returned, it would only be because something else on the system asked for a sync (and it was easier to write everything), or because you're operating on a filesystem that chooses to write data sooner than required. Both of these are basically working by accident (i.e., may well not work on other POSIX systems). I think what you really want is to instruct the kernel that this data should be written out, and you want to block until that happens. That's exactly what fsync() is supposed to do. The fact that some kernels hammer the I/O subsystem while doing so is a bug in those kernels, not the fsync() interface.

ryandrake 13y ago
Coming from a graphics background, this reminds me of a similar problem in graphics. In the OpenGL API, you have two functions (hard core graphics pedants, please forgive my simplification): There's glFinish() which instructs the graphics system to process all graphics commands previously sent and to block until everything is done. Then there's glFlush() which returns immediately, and basically says, make sure all commands previously sent will finish "in a finite amount of time" (as opposed to "maybe never"). From the discussion here, it seems that fsync() is like glFinish(). Maybe what's needed would be something similar to glFlush()? Something that says, "kernel, now would be a good time to start writing out data, but I'm not going to wait."
- dap 13y ago
  That's potentially interesting, but the question is: if you're not going to do anything that relies on the data being on stable storage, why bother asking the system to write it out? Most of the use cases I think of involve a server of some kind servicing client requests. Most of the time, the only sane semantics are that if the request completes successfully, then the change will survive a system crash. In that case, you have to fsync() (or equivalent). Conversely, if the client doesn't need that guarantee (e.g., this is a cache that can be reconstructed from elsewhere, as in the case of a CDN), then there's no reason to make sure it's on stable storage at all. You're basically using the filesystem as a large, slow extension of DRAM, and if everything you ever write fits in DRAM, you wouldn't care if the kernel ever wrote it out. Is there a use case you had in mind? I can't think of a middle ground.
  - Filligree 13y ago
    Yes, a non-blocking fsync. I'd like to ask the kernel to write some data out, but I want to go on servicing other requests without blocking one OS thread entirely. And sometimes I really don't care when data is written out, just that it happens in the right order. I may be okay with losing a few seconds to a minute of work, but not okay with blocking all computation while I'm waiting for fsync.
    - dap 13y ago
      jerf described a similar case where one is okay with losing some data, but not "too much". I'm not sure how to make that approach rigorous. (If rigor isn't important, and it's just best effort with no guarantees at all, then don't bother with fsync at all.) I'm also not sure in what higher-level use case it actually makes sense. Sorry if I'm being thick, but saying "the case where I want exactly that" doesn't help explain that case :) I'm looking for a higher-level description of the problem that would solve (e.g., "an ACID database", or "a CDN" -- except those are cases where you may want zero or any data loss, respectively).
      - Dylan16807 13y ago
        In most things I do on my computer, a few seconds or a couple minutes of data loss is acceptable, but data corruption is not acceptable. Without ordering guarantees data can become corrupt, something from weeks ago can disappear because you tried to update it. that is what is unacceptable. Perhaps compare to the uberblocks on zfs. You can wipe out the most recent one, or the most recent fifty, and still have a consistent, slightly rewound, filesystem. Take that level of consistency and add a notification when everything is written out, and you have a nice nonblocking fsync.
      - nitrogen 13y ago
        The kernel might wait for a very long time before writing any data to disk (especially in laptop mode with a spinning disk drive). Telling it to start writing now seems like a useful operation for an application that can handle the loss of recent writes, but would prefer to have data start going to disk anyway. Otherwise a power failure could mean losing 30 seconds of writes or more.
- jamesaguilar 13y ago
  There is already fflush, and it is generally not what people calling fsync want.
  - zurn 13y ago
    fflush is unrelated to the business of flushing data to disk. It's just a libc FILE* thing. It causes libc to empty the (typically tiny) stdio buffer associated with the FILE* using the write() system call.
    - jamesaguilar 13y ago
      Oops, my bad.
SamReidHughes2 13y ago
What bodyfour is proposing is not just getting a write-notification -- it's in tandem with being able to specify write ordering. If you need to be able to specify write ordering, you also want to be able to have a write notification. fsync does not work because it returns when everything (in the same thread? or whatever) has been written to disk, and doesn't let you wait for a particular block to have been written.
- dap 13y ago
  > What bodyfour is proposing is not just getting a write-notification -- it's in tandem with being able to specify write ordering. If you need to be able to specify write ordering, you also want to be able to have a write notification. It's still not enough: with write ordering and notification but no instruction to actually write the data soon, the kernel can buffer it indefinitely. If you want the data on disk, use fsync(). If you don't care, don't. If the problem is that you can't afford the latency imposed by the multiple fsync() calls required to ensure correct data ordering for your application, fine. But that's not the problem the OP talks about. That was about fsync() hammering the I/O subsystem. You can solve that problem by fixing the fsync() implementation. > fsync does not work because it returns when everything (in the same thread? or whatever) has been written to disk, and doesn't let you wait for a particular block to have been written. If you really want that, could you mmap(2) the file and use msync(2)? For this as well as the other cases described (e.g., wanting write ordering), I don't know what the intended use case actually is, but is it possible that there's another way to organize the data that's still correct and performs well without changing the POSIX interface? It seems likely, given the number of different programs out there that manage to get by with it, and there's a rather significant cost to adding a new interface. (One option is to write everything you need into a temporary file, fsync() it, then synchronously rename it to "commit" it. That still requires two fsync's, but never more than that. You can generalize this for multiple files using a temporary directory.) FWIW, I typically work on illumos systems. On fsync(), ZFS records only an intent log record. That alone helps, since it's not stopping the world to write out everything that's been buffered. For particularly latency-sensitive applications, we use a separate intent log device on an SSD. (Regardless of write ordering and filesystem optimizations, an SSD is necessary in order to guarantee something is on stable storage with latency better than spindles can provide.) This configuration works very well.
bodyfour 13y ago
> you could literally be waiting forever Technically that's true, but it's not an issue in real life. Any sensible operating system will flush dirty buffers to disk eventually. Imagine if a sudden power outage lost writes you had made a month ago; it would be considered a pretty severe OS error. So in reality, you might be waiting many seconds for the write to hit disk, but you won't be waiting hours. > That's exactly what fsync() is supposed to do. No, fsync is "Make sure the last write made it to disk. IN fact, force it out immediately. Also, all of the other pending writes as well, even if I'm not concerned about them. Oh, and block my thread until you're finished." It's an expensive operation.
- otterley 13y ago
  > No, fsync is "Make sure the last write made it to disk. IN fact, force it out immediately. Also, all of the other pending writes as well, even if I'm not concerned about them. ..." POSIX 1003.1 (2004) doesn't go so far as to require unrelated data to be committed (emphasis mine): "The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes." (http://pubs.opengroup.org/onlinepubs/009695299/functions/fsync.html http://pubs.opengroup.org/onlinepubs/009695299/functions/fsy...) I think there's a good argument that Linux and other workalike OSes go too far in flushing all dirty pages to disk when fsync() is called. sync(), on the other hand, does require all dirty pages to be committed, though, ironically, it is permitted to return asynchronously (http://pubs.opengroup.org/onlinepubs/009695299/functions/sync.html http://pubs.opengroup.org/onlinepubs/009695299/functions/syn...).
  - bodyfour 13y ago
    True, and it's good of you to clarify. However, what if the "other pending writes" are to the same file descriptor? Just because I'm eager to know when one write happens doesn't mean I care about other writes I made elsewhere. sync() is basically a relic. Historically it only existed so that a UNIX userland process could roughly control the rate that dirty buffers went to disk. This was the "update" daemon historically, known as "fsflush" on SYSV. At least on linux that's internal to the kernel these days.
riffraff 13y ago
could you explain what the kernels that "do it right" do differently from those that "hammer the I/O subsystem"? Not being an expert, it would seem to me there shouldn't be much difference but obviously there is (i.e. TFA also says "linux woes").
- dap 13y ago
  One of the biggest problems here is that spindles usually don't handle oversubscription well. They service requests quickly up to a certain queue depth, and after that I/O times become significantly longer and more erratic. So if you've got a RAID5 system swamped by a large streaming write workload, an fsync that triggers even a single write may take hundreds to thousands of milliseconds to complete. ZFS does a few things here: - On fsync(), the filesystem only records intent log records to disk. It doesn't record all the filesystem updates needed to implement the change. This cuts back on I/O. - ZFS supports using a separate physical device only for the intent log (called a "slog"). Even if a huge (non-synchronous) workload is slamming most of the spindles, writes to the slog can still complete quickly. This helps even if the slog is another spindle, but it's a huge win if it's an SSD. - ZFS throttles writes to avoid issuing so much to disks concurrently that they start behaving poorly. For more on this (and recent work to improve it), see http://dtrace.org/blogs/ahl/2013/12/27/zfs-fundamentals-the-write-throttle/ http://dtrace.org/blogs/ahl/2013/12/27/zfs-fundamentals-the-.... This may all apply to ZFS on Linux as well.

by dap 13y ago

ryandrake 13y ago

Coming from a graphics background, this reminds me of a similar problem in graphics. In the OpenGL API, you have two functions (hard core graphics pedants, please forgive my simplification): There's glFinish() which instructs the graphics system to process all graphics commands previously sent and to block until everything is done. Then there's glFlush() which returns immediately, and basically says, make sure all commands previously sent will finish "in a finite amount of time" (as opposed to "maybe never"). From the discussion here, it seems that fsync() is like glFinish(). Maybe what's needed would be something similar to glFlush()? Something that says, "kernel, now would be a good time to start writing out data, but I'm not going to wait."

dap 13y ago
That's potentially interesting, but the question is: if you're not going to do anything that relies on the data being on stable storage, why bother asking the system to write it out? Most of the use cases I think of involve a server of some kind servicing client requests. Most of the time, the only sane semantics are that if the request completes successfully, then the change will survive a system crash. In that case, you have to fsync() (or equivalent). Conversely, if the client doesn't need that guarantee (e.g., this is a cache that can be reconstructed from elsewhere, as in the case of a CDN), then there's no reason to make sure it's on stable storage at all. You're basically using the filesystem as a large, slow extension of DRAM, and if everything you ever write fits in DRAM, you wouldn't care if the kernel ever wrote it out. Is there a use case you had in mind? I can't think of a middle ground.
- Filligree 13y ago
  Yes, a non-blocking fsync. I'd like to ask the kernel to write some data out, but I want to go on servicing other requests without blocking one OS thread entirely. And sometimes I really don't care when data is written out, just that it happens in the right order. I may be okay with losing a few seconds to a minute of work, but not okay with blocking all computation while I'm waiting for fsync.
  - dap 13y ago
    jerf described a similar case where one is okay with losing some data, but not "too much". I'm not sure how to make that approach rigorous. (If rigor isn't important, and it's just best effort with no guarantees at all, then don't bother with fsync at all.) I'm also not sure in what higher-level use case it actually makes sense. Sorry if I'm being thick, but saying "the case where I want exactly that" doesn't help explain that case :) I'm looking for a higher-level description of the problem that would solve (e.g., "an ACID database", or "a CDN" -- except those are cases where you may want zero or any data loss, respectively).
    - Dylan16807 13y ago
      In most things I do on my computer, a few seconds or a couple minutes of data loss is acceptable, but data corruption is not acceptable. Without ordering guarantees data can become corrupt, something from weeks ago can disappear because you tried to update it. that is what is unacceptable. Perhaps compare to the uberblocks on zfs. You can wipe out the most recent one, or the most recent fifty, and still have a consistent, slightly rewound, filesystem. Take that level of consistency and add a notification when everything is written out, and you have a nice nonblocking fsync.
    - nitrogen 13y ago
      The kernel might wait for a very long time before writing any data to disk (especially in laptop mode with a spinning disk drive). Telling it to start writing now seems like a useful operation for an application that can handle the loss of recent writes, but would prefer to have data start going to disk anyway. Otherwise a power failure could mean losing 30 seconds of writes or more.
jamesaguilar 13y ago
There is already fflush, and it is generally not what people calling fsync want.
- zurn 13y ago
  fflush is unrelated to the business of flushing data to disk. It's just a libc FILE* thing. It causes libc to empty the (typically tiny) stdio buffer associated with the FILE* using the write() system call.
  - jamesaguilar 13y ago
    Oops, my bad.

SamReidHughes2 13y ago

What bodyfour is proposing is not just getting a write-notification -- it's in tandem with being able to specify write ordering. If you need to be able to specify write ordering, you also want to be able to have a write notification. fsync does not work because it returns when everything (in the same thread? or whatever) has been written to disk, and doesn't let you wait for a particular block to have been written.

dap 13y ago
> What bodyfour is proposing is not just getting a write-notification -- it's in tandem with being able to specify write ordering. If you need to be able to specify write ordering, you also want to be able to have a write notification. It's still not enough: with write ordering and notification but no instruction to actually write the data soon, the kernel can buffer it indefinitely. If you want the data on disk, use fsync(). If you don't care, don't. If the problem is that you can't afford the latency imposed by the multiple fsync() calls required to ensure correct data ordering for your application, fine. But that's not the problem the OP talks about. That was about fsync() hammering the I/O subsystem. You can solve that problem by fixing the fsync() implementation. > fsync does not work because it returns when everything (in the same thread? or whatever) has been written to disk, and doesn't let you wait for a particular block to have been written. If you really want that, could you mmap(2) the file and use msync(2)? For this as well as the other cases described (e.g., wanting write ordering), I don't know what the intended use case actually is, but is it possible that there's another way to organize the data that's still correct and performs well without changing the POSIX interface? It seems likely, given the number of different programs out there that manage to get by with it, and there's a rather significant cost to adding a new interface. (One option is to write everything you need into a temporary file, fsync() it, then synchronously rename it to "commit" it. That still requires two fsync's, but never more than that. You can generalize this for multiple files using a temporary directory.) FWIW, I typically work on illumos systems. On fsync(), ZFS records only an intent log record. That alone helps, since it's not stopping the world to write out everything that's been buffered. For particularly latency-sensitive applications, we use a separate intent log device on an SSD. (Regardless of write ordering and filesystem optimizations, an SSD is necessary in order to guarantee something is on stable storage with latency better than spindles can provide.) This configuration works very well.

bodyfour 13y ago

> you could literally be waiting forever Technically that's true, but it's not an issue in real life. Any sensible operating system will flush dirty buffers to disk eventually. Imagine if a sudden power outage lost writes you had made a month ago; it would be considered a pretty severe OS error. So in reality, you might be waiting many seconds for the write to hit disk, but you won't be waiting hours. > That's exactly what fsync() is supposed to do. No, fsync is "Make sure the last write made it to disk. IN fact, force it out immediately. Also, all of the other pending writes as well, even if I'm not concerned about them. Oh, and block my thread until you're finished." It's an expensive operation.

otterley 13y ago
> No, fsync is "Make sure the last write made it to disk. IN fact, force it out immediately. Also, all of the other pending writes as well, even if I'm not concerned about them. ..." POSIX 1003.1 (2004) doesn't go so far as to require unrelated data to be committed (emphasis mine): "The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes." (http://pubs.opengroup.org/onlinepubs/009695299/functions/fsync.html http://pubs.opengroup.org/onlinepubs/009695299/functions/fsy...) I think there's a good argument that Linux and other workalike OSes go too far in flushing all dirty pages to disk when fsync() is called. sync(), on the other hand, does require all dirty pages to be committed, though, ironically, it is permitted to return asynchronously (http://pubs.opengroup.org/onlinepubs/009695299/functions/sync.html http://pubs.opengroup.org/onlinepubs/009695299/functions/syn...).
- bodyfour 13y ago
  True, and it's good of you to clarify. However, what if the "other pending writes" are to the same file descriptor? Just because I'm eager to know when one write happens doesn't mean I care about other writes I made elsewhere. sync() is basically a relic. Historically it only existed so that a UNIX userland process could roughly control the rate that dirty buffers went to disk. This was the "update" daemon historically, known as "fsflush" on SYSV. At least on linux that's internal to the kernel these days.

riffraff 13y ago

could you explain what the kernels that "do it right" do differently from those that "hammer the I/O subsystem"? Not being an expert, it would seem to me there shouldn't be much difference but obviously there is (i.e. TFA also says "linux woes").

dap 13y ago
One of the biggest problems here is that spindles usually don't handle oversubscription well. They service requests quickly up to a certain queue depth, and after that I/O times become significantly longer and more erratic. So if you've got a RAID5 system swamped by a large streaming write workload, an fsync that triggers even a single write may take hundreds to thousands of milliseconds to complete. ZFS does a few things here: - On fsync(), the filesystem only records intent log records to disk. It doesn't record all the filesystem updates needed to implement the change. This cuts back on I/O. - ZFS supports using a separate physical device only for the intent log (called a "slog"). Even if a huge (non-synchronous) workload is slamming most of the spindles, writes to the slog can still complete quickly. This helps even if the slog is another spindle, but it's a huge win if it's an SSD. - ZFS throttles writes to avoid issuing so much to disks concurrently that they start behaving poorly. For more on this (and recent work to improve it), see http://dtrace.org/blogs/ahl/2013/12/27/zfs-fundamentals-the-write-throttle/ http://dtrace.org/blogs/ahl/2013/12/27/zfs-fundamentals-the-.... This may all apply to ZFS on Linux as well.