5 ms·

That's potentially interesting, but the question is: if you're not going to do anything that relies on the data being on stable storage, why bother asking the s

by dap 13y ago

That's potentially interesting, but the question is: if you're not going to do anything that relies on the data being on stable storage, why bother asking the system to write it out? Most of the use cases I think of involve a server of some kind servicing client requests. Most of the time, the only sane semantics are that if the request completes successfully, then the change will survive a system crash. In that case, you have to fsync() (or equivalent). Conversely, if the client doesn't need that guarantee (e.g., this is a cache that can be reconstructed from elsewhere, as in the case of a CDN), then there's no reason to make sure it's on stable storage at all. You're basically using the filesystem as a large, slow extension of DRAM, and if everything you ever write fits in DRAM, you wouldn't care if the kernel ever wrote it out. Is there a use case you had in mind? I can't think of a middle ground.

Filligree 13y ago
Yes, a non-blocking fsync. I'd like to ask the kernel to write some data out, but I want to go on servicing other requests without blocking one OS thread entirely. And sometimes I really don't care when data is written out, just that it happens in the right order. I may be okay with losing a few seconds to a minute of work, but not okay with blocking all computation while I'm waiting for fsync.
- dap 13y ago
  jerf described a similar case where one is okay with losing some data, but not "too much". I'm not sure how to make that approach rigorous. (If rigor isn't important, and it's just best effort with no guarantees at all, then don't bother with fsync at all.) I'm also not sure in what higher-level use case it actually makes sense. Sorry if I'm being thick, but saying "the case where I want exactly that" doesn't help explain that case :) I'm looking for a higher-level description of the problem that would solve (e.g., "an ACID database", or "a CDN" -- except those are cases where you may want zero or any data loss, respectively).
  - Dylan16807 13y ago
    In most things I do on my computer, a few seconds or a couple minutes of data loss is acceptable, but data corruption is not acceptable. Without ordering guarantees data can become corrupt, something from weeks ago can disappear because you tried to update it. that is what is unacceptable. Perhaps compare to the uberblocks on zfs. You can wipe out the most recent one, or the most recent fifty, and still have a consistent, slightly rewound, filesystem. Take that level of consistency and add a notification when everything is written out, and you have a nice nonblocking fsync.
  - nitrogen 13y ago
    The kernel might wait for a very long time before writing any data to disk (especially in laptop mode with a spinning disk drive). Telling it to start writing now seems like a useful operation for an application that can handle the loss of recent writes, but would prefer to have data start going to disk anyway. Otherwise a power failure could mean losing 30 seconds of writes or more.
jerf 13y ago
Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. In fact filesystems in general attempt to implement this for themselves, because a filesystem should ideally always be in a consistent state... it may not be the right state, per se, but it should not be actively inconsistent, leaving you (or fsck) to basically guess what the correct state is. The problem appears to be that today there's only "write this all out now and DO ABSOLUTELY NOTHING ELSE until that happens", and "yeah, whatever, write it whatever order you like and I sure hope it all works out." Is that correct? There really isn't anything like a write barrier? All my reading of the links here seem to indicate that but I find it hard to believe that really is an accurate summarization of the current state of affairs on Linux. (Though I concede that I can see how hard it would be to propagate such a guarantee all the way from the disk hardware, through the drivers, through a large and varied number of file systems, all the way out to user space, without bugs, bugs, bugs.)
- dap 13y ago
  > Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. So a little bit of data loss is okay, but a lot isn't? How does a program or operator determine how much is okay and how much isn't? How does the application ensure that that limit isn't exceeded? Without answers to these questions, it feels like "you'll probably be fine, but I can't be sure of anything", which feels pretty lame. But if a little data loss really is okay, then forget about both ordering and fsync and truncate the log after the last consecutive valid record. > The problem appears to be that today there's only "write this all out now and DO ABSOLUTELY NOTHING ELSE until that happens", and "yeah, whatever, write it whatever order you like and I sure hope it all works out." > Is that correct? I don't think so, but it depends on what you mean by "absolutely nothing else". You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on. > There really isn't anything like a write barrier? Other than fsync() and equivalents, I don't know of one. Non-blocking write barriers would represent a much more complicated abstraction for both applications and the filesystem, and (as you can tell from my comments on this thread) I'm not convinced it's worth the complexity for any rigorous program.

by dap 13y ago

Filligree 13y ago

Yes, a non-blocking fsync. I'd like to ask the kernel to write some data out, but I want to go on servicing other requests without blocking one OS thread entirely. And sometimes I really don't care when data is written out, just that it happens in the right order. I may be okay with losing a few seconds to a minute of work, but not okay with blocking all computation while I'm waiting for fsync.

dap 13y ago
jerf described a similar case where one is okay with losing some data, but not "too much". I'm not sure how to make that approach rigorous. (If rigor isn't important, and it's just best effort with no guarantees at all, then don't bother with fsync at all.) I'm also not sure in what higher-level use case it actually makes sense. Sorry if I'm being thick, but saying "the case where I want exactly that" doesn't help explain that case :) I'm looking for a higher-level description of the problem that would solve (e.g., "an ACID database", or "a CDN" -- except those are cases where you may want zero or any data loss, respectively).
- Dylan16807 13y ago
  In most things I do on my computer, a few seconds or a couple minutes of data loss is acceptable, but data corruption is not acceptable. Without ordering guarantees data can become corrupt, something from weeks ago can disappear because you tried to update it. that is what is unacceptable. Perhaps compare to the uberblocks on zfs. You can wipe out the most recent one, or the most recent fifty, and still have a consistent, slightly rewound, filesystem. Take that level of consistency and add a notification when everything is written out, and you have a nice nonblocking fsync.
- nitrogen 13y ago
  The kernel might wait for a very long time before writing any data to disk (especially in laptop mode with a spinning disk drive). Telling it to start writing now seems like a useful operation for an application that can handle the loss of recent writes, but would prefer to have data start going to disk anyway. Otherwise a power failure could mean losing 30 seconds of writes or more.

jerf 13y ago

Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. In fact filesystems in general attempt to implement this for themselves, because a filesystem should ideally always be in a consistent state... it may not be the right state, per se, but it should not be actively inconsistent, leaving you (or fsck) to basically guess what the correct state is. The problem appears to be that today there's only "write this all out now and DO ABSOLUTELY NOTHING ELSE until that happens", and "yeah, whatever, write it whatever order you like and I sure hope it all works out." Is that correct? There really isn't anything like a write barrier? All my reading of the links here seem to indicate that but I find it hard to believe that really is an accurate summarization of the current state of affairs on Linux. (Though I concede that I can see how hard it would be to propagate such a guarantee all the way from the disk hardware, through the drivers, through a large and varied number of file systems, all the way out to user space, without bugs, bugs, bugs.)

dap 13y ago
> Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. So a little bit of data loss is okay, but a lot isn't? How does a program or operator determine how much is okay and how much isn't? How does the application ensure that that limit isn't exceeded? Without answers to these questions, it feels like "you'll probably be fine, but I can't be sure of anything", which feels pretty lame. But if a little data loss really is okay, then forget about both ordering and fsync and truncate the log after the last consecutive valid record. > The problem appears to be that today there's only "write this all out now and DO ABSOLUTELY NOTHING ELSE until that happens", and "yeah, whatever, write it whatever order you like and I sure hope it all works out." > Is that correct? I don't think so, but it depends on what you mean by "absolutely nothing else". You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on. > There really isn't anything like a write barrier? Other than fsync() and equivalents, I don't know of one. Non-blocking write barriers would represent a much more complicated abstraction for both applications and the filesystem, and (as you can tell from my comments on this thread) I'm not convinced it's worth the complexity for any rigorous program.