5 ms·
Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written
by jerf 13y ago
Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general.
In fact filesystems in general attempt to implement this for themselves, because a filesystem should ideally always be in a consistent state... it may not be the right state, per se, but it should not be actively inconsistent, leaving you (or fsck) to basically guess what the correct state is.
The problem appears to be that today there's only "write this all out now and DO ABSOLUTELY NOTHING ELSE until that happens", and "yeah, whatever, write it whatever order you like and I sure hope it all works out."
Is that correct? There really isn't anything like a write barrier? All my reading of the links here seem to indicate that but I find it hard to believe that really is an accurate summarization of the current state of affairs on Linux. (Though I concede that I can see how hard it would be to propagate such a guarantee all the way from the disk hardware, through the drivers, through a large and varied number of file systems, all the way out to user space, without bugs, bugs, bugs.)
- dap 13y ago> Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. So a little bit of data loss is okay, but a lot isn't? How does a program or operator determine how much is okay and how much isn't? How does the application ensure that that limit isn't exceeded? Without answers to these questions, it feels like "you'll probably be fine, but I can't be sure of anything", which feels pretty lame. But if a little data loss really is okay, then forget about both ordering and fsync and truncate the log after the last consecutive valid record. > The problem appears to be that today there's only "write this all out now and DO ABSOLUTELY NOTHING ELSE until that happens", and "yeah, whatever, write it whatever order you like and I sure hope it all works out." > Is that correct? I don't think so, but it depends on what you mean by "absolutely nothing else". You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on. > There really isn't anything like a write barrier? Other than fsync() and equivalents, I don't know of one. Non-blocking write barriers would represent a much more complicated abstraction for both applications and the filesystem, and (as you can tell from my comments on this thread) I'm not convinced it's worth the complexity for any rigorous program.
- maxlybbert 13y agoI've never implemented database transactions, so this might be a little hand wavy. A database transaction promises that all changes happen or no changes happen, i.e., that you don't get some of the changes. The easiest way to implement that is to write what you're going to do to a log, and then once you know that the log is on disk, you make the change in the actual data file. Two phase commit. But what if I could loosen that some? It isn't that important when the log file is actually written; only that it will hit disk before the changes to the data file. If I write to the log, tell the operating system "after you get around to putting that on disk, make these changes to this other file," I can promise that the transaction will be atomic. I can't tell you if the transaction will actually be committed, but I know there won't be a partial commit. If the system crashes before the log gets updated, the transaction doesn't commit. But I don't have to wait around to find out when the file gets written out. I just need to know that if anything gets written to the data file, then all my writes to the log file are safely on disk.
- rosser 13y agoYou can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on. No, you can't. At least not without knowing what you're doing and careful planning. My day job is PostgreSQL DBA, and I've been doing that for most of a decade now. As the kids on the Reddits would say, "I've seen some shit." I have some rather large servers, with some rather powerful IO subsystems — my production environment has SAS SLC SSDs under hardware RAID with a ginormous cache. I still see the behavior described in TFA far more often than I'd like. Linux really is pretty dumb here. For example, because of this fsync() issue, and the fact that fsync() calls flush all outstanding writes for the entire filesystem the file(s) being fsync()'ed reside upon, I've set up my servers such that my $PGDATA/pg_xlog directory is a symlink from the volume mounted at (well, above) $PGDATA to a separate, much smaller filesystem. (That is: transaction logs, which must be fsync()'ed often to guarantee consistency, and enable crash recovery, reside on a smaller, dedicated filesystem, separate from the rest of my database's disk footprint.) If I didn't do that, at every checkpoint, my performance would measurably fall. I learned this lesson the hard way, at an old job, where my postgres clusters lived on a SAN — it wasn't just my db instances that were being adversely affected by this IO storm. It was everything else that lived on the filer, too. That's how bad it can be.