3 ms·
You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on. No, you can't. A
by rosser 13y ago
You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on.
No, you can't. At least not without knowing what you're doing and careful planning.
My day job is PostgreSQL DBA, and I've been doing that for most of a decade now. As the kids on the Reddits would say, "I've seen some shit." I have some rather large servers, with some rather powerful IO subsystems — my production environment has SAS SLC SSDs under hardware RAID with a ginormous cache. I still see the behavior described in TFA far more often than I'd like. Linux really is pretty dumb here.
For example, because of this fsync() issue, and the fact that fsync() calls flush all outstanding writes for the entire filesystem the file(s) being fsync()'ed reside upon, I've set up my servers such that my $PGDATA/pg_xlog directory is a symlink from the volume mounted at (well, above) $PGDATA to a separate, much smaller filesystem. (That is: transaction logs, which must be fsync()'ed often to guarantee consistency, and enable crash recovery, reside on a smaller, dedicated filesystem, separate from the rest of my database's disk footprint.)
If I didn't do that, at every checkpoint, my performance would measurably fall. I learned this lesson the hard way, at an old job, where my postgres clusters lived on a SAN — it wasn't just my db instances that were being adversely affected by this IO storm. It was everything else that lived on the filer, too.
That's how bad it can be.
- dap 13y agoYou're absolutely right. The situation on Linux sounds bad, and it sounds like work's being done to improve that. Some other systems handle this case fine, though, and the problem isn't with the fsync() interface. Aside: I worked on one of those filers, and I've seen some shit too. :) Firmware, especially disk firmware, was the worst.
- tytso 13y agoIt's not true that fsync() calls flush all outstanding writes for the entire file system; that was true for ext3 in data=ordered mode, but it's definitely not true for ext4 or xfs. If you use fdatasync(), and there were no write commands that issued against the file descriptor that required metadata updates (i.e., you didn't do any block allocations, etc), then both ext4 and xfs won't need to trigger a journal commit, so the only thing that has to get sent to disk is all of the dirty metadata blocks, followed by a SYNC CACHE command which forces the disk drive to guarantee that all writes sent to the disk will survive a power cut. If you use fsync() and/or you have allocated blocks or otherwise performanced a write which required updating file system metadata, and thus will require a journal commit, then you will need to force out all pending metadata updates to the journal as part of the file system commit, but that's still not the same as "flush all outstanding writes for the entire file system".
- bodyfour 13y ago> or you have allocated blocks or otherwise performanced a write which required updating file system metadata What if you're appending to a file, and want to checkpoint every so often. I guess you can be clever with fallocate(FALLOC_FL_KEEP_SIZE) to avoid the block allocation, but won't st_size still need to be updated? I also assume that st_mtime doesn't count towards dirtying the metadata.