13 ms·

I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hi

I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words, what you want should be nearly free, but instead you have to do a very expensive operation. For an example for notification: suppose I have a temporary file with data that is being journaled into a data store. The operation I want to do is: 1. Apply changes to the store 2. Wait until all of those writes hit disk 3. Delete the temporary file I don't care if step 2 takes 5 minutes, nor do I want the kernel to schedule my writes in any particular way. If you implement step 2 as a fsync() (or fdatasync()) you're having a potentially huge impact on I/O throughput. I've seen these frequent fsync()s cause 50x performance drops!

Qantourisc 13y ago
Not a kernel dev but to my knowledge "write B must hit disk after write A" would actually be rather easy to implement: if the storage device supports it: insert a write barrier. For the "let me know when write A is on disk" adding a callback to a write-barrier would do, not sure what the kernel queue looks like though.
- Filligree 13y ago
  In principle, yes. I believe most modern filesystems use (kernel-level) write barriers to ensure filesystem consistency, and those do depend on having hardware barriers as well. The kernel is not set up to export this interface, however, and there doesn't appear to be any serious work being done to fix that. EDIT: http://pl.atyp.us/2013-11-fixing-fsync.html http://pl.atyp.us/2013-11-fixing-fsync.html has more details.
- deathanatos 13y ago
  While a "notify me when this write completes" (but don't feel the need to rush) might be useful, I think it'd be more useful to be able to push both write A and write B to the kernel, along with the condition of "A before B". You not only get the write ordering, but the kernel gets full knowledge of what's going on. Perhaps it could use this to write A and B together (but journaled at the FS level) to guarantee the order, whereas a simple write-complete notification implies at least two writes, and the latency of doing such.
  - bodyfour 13y ago
    The two-write operation you propose is simpler, but you're losing a lot of generality. Specifically, think about: * I want to notify a network client that its write is done * When this write is done, I want to delete a temporary file etc
    - deathanatos 13y ago
      Good points — there's definitely a place for a write barrier. It may be that a simple write barrier, while not as efficient for this use case, might be generally more useful. For your second point, if you take my initial "write B depends on the completion of write A", and extend it allow arbitrary kernel commands to depend on others, then you could still do that, (delete B depends on write A), however, things are probably too complex at this point, and a simple write barrier is better.
mnarayan01 13y ago
Is that not the purpose of [aio_fsync](http://pubs.opengroup.org/onlinepubs/009696899/functions/aio_fsync.html http://pubs.opengroup.org/onlinepubs/009696899/functions/aio...)? Disclaimer: I've never used it, I just filed it in the back of my mind for if I wanted to do basically what you just described.
- justincormack 13y ago
  Doesn't exist on Linux as a kernel interface, although the API is there; if it works it is either a thread doing fsync or its synchronous.
jcampbell1 13y ago
I don't understand, but I am not very knowledgable about this any of this. My understanding is that fsync is an instruction to command the physical disk to flush its write buffer to the non-volatile (platters/flash) portion of storage, so it is safe to yank the cord from the wall. In your scenario, it seems like you could have a situation where the temp file and the data are sitting on RAM inside the HD. I honestly don't see how multiple processes that share a single physical disk can't get trashed by a processes that instructs the disk to constantly flush the buffer. It seems like asking for a network interface that doesn't slow down the network due other processes.
- nknighthb 13y ago
  There are multiple levels of caching. The cache in a disk is just one. Filesystem cache is another, which happens in RAM. There may be a layer of caching between filesystem and disk. Applications routinely cache data in their own memory allocations before writing to the filesystem. fsync lives between applications and the physical disk. Its (intended) effect is to flush dirty data that an application has written to the filesystem all the way to the physical disk. For maximum performance and minimum fragmentation, you want to wait as long as possible before data is flushed from the filesystem's cache. In a typical storage system, the filesystem is what decides the approximate ordering of writes and the allocation of disk space. Disks have no idea what's going on, they just respond to commands to write certain data to certain locations; their discretion is minimal, at most they will re-order a small batch of writes for optimum head movement. Delaying a flush as long as possible allows the filesystem to coalesce multiple writes to the same location into one write, allocate disk space in the largest possible chunks in the best locations to avoid fragmentation and excessive seeking, order the writes for rough optimization and for data and metadata consistency. Caching data in RAM is a good thing. fsync is what you use when you've got important data that needs to be preserved immediately. Most data on a typical PC or server is just not that important. You're better off letting the filesystem do its job. And there's a whole continuum of data consistency in between, where you might want certain operations to happen in a certain order, or data to be written in a certain order if it's written, even if you don't necessarily need them to end up on the disk right now.
dap 13y ago
> In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. If you don't issue an fsync(), the kernel never has to write anything. If you had a different function that did what you suggest, returning when the kernel had decided to write the data (without instructing the kernel to do so soonish), you could literally be waiting forever. If the function ever returned, it would only be because something else on the system asked for a sync (and it was easier to write everything), or because you're operating on a filesystem that chooses to write data sooner than required. Both of these are basically working by accident (i.e., may well not work on other POSIX systems). I think what you really want is to instruct the kernel that this data should be written out, and you want to block until that happens. That's exactly what fsync() is supposed to do. The fact that some kernels hammer the I/O subsystem while doing so is a bug in those kernels, not the fsync() interface.
- ryandrake 13y ago
  Coming from a graphics background, this reminds me of a similar problem in graphics. In the OpenGL API, you have two functions (hard core graphics pedants, please forgive my simplification): There's glFinish() which instructs the graphics system to process all graphics commands previously sent and to block until everything is done. Then there's glFlush() which returns immediately, and basically says, make sure all commands previously sent will finish "in a finite amount of time" (as opposed to "maybe never"). From the discussion here, it seems that fsync() is like glFinish(). Maybe what's needed would be something similar to glFlush()? Something that says, "kernel, now would be a good time to start writing out data, but I'm not going to wait."
  - dap 13y ago
    That's potentially interesting, but the question is: if you're not going to do anything that relies on the data being on stable storage, why bother asking the system to write it out? Most of the use cases I think of involve a server of some kind servicing client requests. Most of the time, the only sane semantics are that if the request completes successfully, then the change will survive a system crash. In that case, you have to fsync() (or equivalent). Conversely, if the client doesn't need that guarantee (e.g., this is a cache that can be reconstructed from elsewhere, as in the case of a CDN), then there's no reason to make sure it's on stable storage at all. You're basically using the filesystem as a large, slow extension of DRAM, and if everything you ever write fits in DRAM, you wouldn't care if the kernel ever wrote it out. Is there a use case you had in mind? I can't think of a middle ground.
sanxiyn 13y ago
Featherstitch http://lwn.net/Articles/354861/ http://lwn.net/Articles/354861/ is an interface (and an implementation) for write ordering without blocking. As far as I can tell, it went nowhere.

Qantourisc 13y ago

Not a kernel dev but to my knowledge "write B must hit disk after write A" would actually be rather easy to implement: if the storage device supports it: insert a write barrier. For the "let me know when write A is on disk" adding a callback to a write-barrier would do, not sure what the kernel queue looks like though.

Filligree 13y ago
In principle, yes. I believe most modern filesystems use (kernel-level) write barriers to ensure filesystem consistency, and those do depend on having hardware barriers as well. The kernel is not set up to export this interface, however, and there doesn't appear to be any serious work being done to fix that. EDIT: http://pl.atyp.us/2013-11-fixing-fsync.html http://pl.atyp.us/2013-11-fixing-fsync.html has more details.
deathanatos 13y ago
While a "notify me when this write completes" (but don't feel the need to rush) might be useful, I think it'd be more useful to be able to push both write A and write B to the kernel, along with the condition of "A before B". You not only get the write ordering, but the kernel gets full knowledge of what's going on. Perhaps it could use this to write A and B together (but journaled at the FS level) to guarantee the order, whereas a simple write-complete notification implies at least two writes, and the latency of doing such.
- bodyfour 13y ago
  The two-write operation you propose is simpler, but you're losing a lot of generality. Specifically, think about: * I want to notify a network client that its write is done * When this write is done, I want to delete a temporary file etc
  - deathanatos 13y ago
    Good points — there's definitely a place for a write barrier. It may be that a simple write barrier, while not as efficient for this use case, might be generally more useful. For your second point, if you take my initial "write B depends on the completion of write A", and extend it allow arbitrary kernel commands to depend on others, then you could still do that, (delete B depends on write A), however, things are probably too complex at this point, and a simple write barrier is better.

mnarayan01 13y ago

Is that not the purpose of [aio_fsync](http://pubs.opengroup.org/onlinepubs/009696899/functions/aio_fsync.html http://pubs.opengroup.org/onlinepubs/009696899/functions/aio...)? Disclaimer: I've never used it, I just filed it in the back of my mind for if I wanted to do basically what you just described.

justincormack 13y ago
Doesn't exist on Linux as a kernel interface, although the API is there; if it works it is either a thread doing fsync or its synchronous.

jcampbell1 13y ago

I don't understand, but I am not very knowledgable about this any of this. My understanding is that fsync is an instruction to command the physical disk to flush its write buffer to the non-volatile (platters/flash) portion of storage, so it is safe to yank the cord from the wall. In your scenario, it seems like you could have a situation where the temp file and the data are sitting on RAM inside the HD. I honestly don't see how multiple processes that share a single physical disk can't get trashed by a processes that instructs the disk to constantly flush the buffer. It seems like asking for a network interface that doesn't slow down the network due other processes.

nknighthb 13y ago
There are multiple levels of caching. The cache in a disk is just one. Filesystem cache is another, which happens in RAM. There may be a layer of caching between filesystem and disk. Applications routinely cache data in their own memory allocations before writing to the filesystem. fsync lives between applications and the physical disk. Its (intended) effect is to flush dirty data that an application has written to the filesystem all the way to the physical disk. For maximum performance and minimum fragmentation, you want to wait as long as possible before data is flushed from the filesystem's cache. In a typical storage system, the filesystem is what decides the approximate ordering of writes and the allocation of disk space. Disks have no idea what's going on, they just respond to commands to write certain data to certain locations; their discretion is minimal, at most they will re-order a small batch of writes for optimum head movement. Delaying a flush as long as possible allows the filesystem to coalesce multiple writes to the same location into one write, allocate disk space in the largest possible chunks in the best locations to avoid fragmentation and excessive seeking, order the writes for rough optimization and for data and metadata consistency. Caching data in RAM is a good thing. fsync is what you use when you've got important data that needs to be preserved immediately. Most data on a typical PC or server is just not that important. You're better off letting the filesystem do its job. And there's a whole continuum of data consistency in between, where you might want certain operations to happen in a certain order, or data to be written in a certain order if it's written, even if you don't necessarily need them to end up on the disk right now.

dap 13y ago

> In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. If you don't issue an fsync(), the kernel never has to write anything. If you had a different function that did what you suggest, returning when the kernel had decided to write the data (without instructing the kernel to do so soonish), you could literally be waiting forever. If the function ever returned, it would only be because something else on the system asked for a sync (and it was easier to write everything), or because you're operating on a filesystem that chooses to write data sooner than required. Both of these are basically working by accident (i.e., may well not work on other POSIX systems). I think what you really want is to instruct the kernel that this data should be written out, and you want to block until that happens. That's exactly what fsync() is supposed to do. The fact that some kernels hammer the I/O subsystem while doing so is a bug in those kernels, not the fsync() interface.

ryandrake 13y ago
Coming from a graphics background, this reminds me of a similar problem in graphics. In the OpenGL API, you have two functions (hard core graphics pedants, please forgive my simplification): There's glFinish() which instructs the graphics system to process all graphics commands previously sent and to block until everything is done. Then there's glFlush() which returns immediately, and basically says, make sure all commands previously sent will finish "in a finite amount of time" (as opposed to "maybe never"). From the discussion here, it seems that fsync() is like glFinish(). Maybe what's needed would be something similar to glFlush()? Something that says, "kernel, now would be a good time to start writing out data, but I'm not going to wait."
- dap 13y ago
  That's potentially interesting, but the question is: if you're not going to do anything that relies on the data being on stable storage, why bother asking the system to write it out? Most of the use cases I think of involve a server of some kind servicing client requests. Most of the time, the only sane semantics are that if the request completes successfully, then the change will survive a system crash. In that case, you have to fsync() (or equivalent). Conversely, if the client doesn't need that guarantee (e.g., this is a cache that can be reconstructed from elsewhere, as in the case of a CDN), then there's no reason to make sure it's on stable storage at all. You're basically using the filesystem as a large, slow extension of DRAM, and if everything you ever write fits in DRAM, you wouldn't care if the kernel ever wrote it out. Is there a use case you had in mind? I can't think of a middle ground.

sanxiyn 13y ago

Featherstitch http://lwn.net/Articles/354861/ http://lwn.net/Articles/354861/ is an interface (and an implementation) for write ordering without blocking. As far as I can tell, it went nowhere.