4 ms·

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2): If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing" means to create a _new_ file, writing it, syncing it, and then replacing the old version with rename(2) (and fsyncing the directory). > - "Temp" files always go away completely after a crash. This should be the default for designated temp directories. You can have that: Just don't link the inode of the file. On Linux, you can use O_TMPFILE. Alternatively, just create the file, and keep it open but immediately unlink it (it's a little hack but nothing dramatic). You can also write to /tmp. Normally that gets cleaned at reboot, which might be enough. > - "Log" files can only be appended. In open(2), look for O_APPEND. > Writers cannot seek. That's not how you write a kernel API. If you don't want to seek, then don't. > The file system guarantees that after a crash, the end of the file is at the end of some write How do you intend to implement that? Disks don't support arbitrarily large atomic commits. Making such a guarantee in software is necessarily inefficient. The kernel will not choose your tradeoffs for you. You can do it from userland. Decide yourself what is the least bad way to do it. > "Managed" files are for structured databases. [..] The writer gets a callback when the write has committed to disk. The file system must guarantee that a write for which the callback has been made will survive a crash. Fsync does just that, except it isn't asynchronous. Maybe there's something behind aio(7)?

skybrian 11y ago
Your response pretty aptly demonstrates the problem in the original article. If you write to a temp file and then use rename, you need to pick a suitable temporary name (without a race condition) that other programs will ignore, and now you've probably created a garbage file if the program crashes before you get to the rename. You also need to make sure you've preserved the permissions in the original file, and hope that nobody else is using the same trick to update the same file. There are API's and standard tricks for these things too. But it's difficult to be sure that you've covered all the corner cases and impossible to be sure that everyone else has covered all the corner places. This all adds complexity compared to having a simple, standard, guaranteed way to atomically update a file and makes it less likely that most programs will do it right.
- jstimpfle 11y ago
  As I said, if it's too unbearable a hack for you to unlink() the file directly after creat()ing it (to get an anonymous file handle that nobody else can see), for the unlikely case that the system crashes just in between, then just go ahead and use O_TMPFILE. It was made for you! If you don't want to use O_TMPFILE (not super-portable) then just open(path, O_CREAT|O_EXCL). That fails should there be another file lying around, and you can try again with another random name. Btw. you can also just mmap(2) some anonymous non-disk-backed memory, and possibly open_memstream(3) on it to get a FILE * if that suits you better.
  - colanderman 11y ago
    Even easier than open(2) with a random file name is mkstemp(3).
- Animats 11y ago
  Exactly. It takes a whole collection of workarounds to get well behaved file semantics across crashes, and the workarounds are different for Linux, BSD, and various other UNIX variants, and may differ depending on the file system type and system version.
colanderman 11y ago
Don't know why you're downvoted. Everything you said is correct. Additionally, in Linux, you can get asynchronous fdatasync(2) with sync_file_range(2). aio(7) is weird. It really only works with direct I/O (which is necessary much less often than people think it is), and IIRC aio_fsync(2) isn't implemented on Linux, but that doesn't matter so much because generally direct I/O implies synchronous I/O (but not always). See http://lse.sourceforge.net/io/aio.html http://lse.sourceforge.net/io/aio.html
- ryao 11y ago
  AIO does not require O_DIRECT. Most Linux file systems do not implement aio_fsync because they put the writes into the page cache and block until it is flushed to disk before signaling completion. That means aio_fsync can be implemented by no-op.
  - colanderman 11y ago
    Unfortunately that is not documented in any man page I've read, because that is very useful (crucial) to know and not at all obvious. I reiterate that aio is weird :) Also, I'm pretty sure that issuing IOCB_CMD_FSYNC from io_submit(2) has given me an error in the past. Not sure about aio_fsync(2), because last time I checked, POSIX aio only had a pthread-based backend, rather than the native support that libaio (i.e. io_submit(2)) provides.
Animats 11y ago
Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.) O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly. How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. The file size update can be deferred during heavy write traffic, but you should always get a file size that ends at a write boundary. "fsync" synchs the whole file, not just one I/O, which can take a while. Databases such as MySQL's InnoDB, which puts multiple tables in one file, can have independent I/O going on in different parts of a file. aio(7) has the right mechanism, a callback/signal on completion. But it's not clear if the file system guarantees the data is safely on disk when the completion signal comes in. The original article complains that UNIX/Linux file system semantics aren't well enough defined for database safety. He's right. They're close, but not quite there, because behavior after a crash is unspecified.
- colanderman 11y ago
  sync_file_range(2) allows you to specify ranges to sync, instead of the whole file, but it's not that useful anyway. In most DBs, you have separate data and log. The log is always only appended to, and the nature of disks is such that grouping concurrent transactions together into a single commit is a big win regardless of how syncing works. So syncing all outstanding I/O to the log file is generally what you want to do anyway. While the data store is always being written to, its writes do not need to be synced to disk until a log checkpoint is made (and the old log subsequently discarded). And since that is an intermittent asynchronous process, there is not much to gain from finer-grained syncing here either. See e.g. http://www.postgresql.org/message-id/4A51CB76.5020407@enterprisedb.com http://www.postgresql.org/message-id/4A51CB76.5020407@enterp...
  - anarazel 11y ago
    Actually, we made postgres use sync_file_range() for checkpointing if available in 9.6. Not for durability - there's a few to many caveats in the manpage - but to control how much work a later fsync() has to do. In many workloads checkpointing can generate a lot of writes, and the OS's writeback caching of those can generate a lot of dirty buffers in the kernel's page cache. If the kernel decides to flush those (on it's own or due to an fsync), latency for every other FS operation can skyrocket. We've seen stalls in the 10s of minutes. So we now regularly sync_file_range(SYNC_FILE_RANGE_WRITE), unless the feature is disabled of course, to control how much dirty data the kernel has. See http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=428b1d6b29ca599c5700d4bc4f4ce4c5880369bf http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;... and http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=9cd00c457e6a1ebb984167ac556a9961812a683c http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;...
dspeyer 11y ago
Rename doesn't always do what we want. The one serious investigation I've seen [1] unhelpfully states that "[W]hen a file is appended, and then renamed ... many file systems recognize it and allocate blocks immediately." Suggesting that some file systems don't, but failing to name them. In general, appends followed by directory operations are not order-preserved. [1] https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14...
- jstimpfle 11y ago
  There's no order preserving guarantee. Code should do fd <- open(tmpfile) write(fd, data) fsync(fd) rename(tmpfile, finalpath) fsync(parent(finalpath)) (Strictly speaking there is a problem in the rename because we can't be sure "tmpfile" still refers to the same inode. It would be nice to have a way to link fds, but I'm not aware of any). According to the paper, existing file systems recognize the (open,) write, rename pattern and don't reorder write and rename. That would break too much existing code. But these are filesystem-specific design decisions. Btw. it's just the same with CPUs writing to memory. Generally writes can be reordered if the behaviour is not visible to the executing thread. However if there are multiple threads you might need to insert (CPU-specific) memory barriers to prevent reorderings that are visible to other threads. x86 is like those file systems here in that most incorrect code still works, because x86 makes some ordering guarantees.
caf 11y ago
rename(2) atomic replacement only means atomic with respect to processes running on the system at the same time: anyone looking up the newpath will get either the old file or the new file (and won't, for example, see a spurious ENOENT). It doesn't mean atomic with respect to a system crash. The easiest way to get asynchronous fsync(2) is to do the fsync(2) in a different thread.
- jstimpfle 11y ago
  > The easiest way to get asynchronous fsync(2) is to do the fsync(2) in a different thread. However you would need to create a new thread for each simultaneous asynchronous fsync to get the benefits of asynchronicity.

skybrian 11y ago

Your response pretty aptly demonstrates the problem in the original article. If you write to a temp file and then use rename, you need to pick a suitable temporary name (without a race condition) that other programs will ignore, and now you've probably created a garbage file if the program crashes before you get to the rename. You also need to make sure you've preserved the permissions in the original file, and hope that nobody else is using the same trick to update the same file. There are API's and standard tricks for these things too. But it's difficult to be sure that you've covered all the corner cases and impossible to be sure that everyone else has covered all the corner places. This all adds complexity compared to having a simple, standard, guaranteed way to atomically update a file and makes it less likely that most programs will do it right.

jstimpfle 11y ago
As I said, if it's too unbearable a hack for you to unlink() the file directly after creat()ing it (to get an anonymous file handle that nobody else can see), for the unlikely case that the system crashes just in between, then just go ahead and use O_TMPFILE. It was made for you! If you don't want to use O_TMPFILE (not super-portable) then just open(path, O_CREAT|O_EXCL). That fails should there be another file lying around, and you can try again with another random name. Btw. you can also just mmap(2) some anonymous non-disk-backed memory, and possibly open_memstream(3) on it to get a FILE * if that suits you better.
- colanderman 11y ago
  Even easier than open(2) with a random file name is mkstemp(3).
Animats 11y ago
Exactly. It takes a whole collection of workarounds to get well behaved file semantics across crashes, and the workarounds are different for Linux, BSD, and various other UNIX variants, and may differ depending on the file system type and system version.

colanderman 11y ago

Don't know why you're downvoted. Everything you said is correct. Additionally, in Linux, you can get asynchronous fdatasync(2) with sync_file_range(2). aio(7) is weird. It really only works with direct I/O (which is necessary much less often than people think it is), and IIRC aio_fsync(2) isn't implemented on Linux, but that doesn't matter so much because generally direct I/O implies synchronous I/O (but not always). See http://lse.sourceforge.net/io/aio.html http://lse.sourceforge.net/io/aio.html

ryao 11y ago
AIO does not require O_DIRECT. Most Linux file systems do not implement aio_fsync because they put the writes into the page cache and block until it is flushed to disk before signaling completion. That means aio_fsync can be implemented by no-op.
- colanderman 11y ago
  Unfortunately that is not documented in any man page I've read, because that is very useful (crucial) to know and not at all obvious. I reiterate that aio is weird :) Also, I'm pretty sure that issuing IOCB_CMD_FSYNC from io_submit(2) has given me an error in the past. Not sure about aio_fsync(2), because last time I checked, POSIX aio only had a pthread-based backend, rather than the native support that libaio (i.e. io_submit(2)) provides.

Animats 11y ago

Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.) O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly. How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. The file size update can be deferred during heavy write traffic, but you should always get a file size that ends at a write boundary. "fsync" synchs the whole file, not just one I/O, which can take a while. Databases such as MySQL's InnoDB, which puts multiple tables in one file, can have independent I/O going on in different parts of a file. aio(7) has the right mechanism, a callback/signal on completion. But it's not clear if the file system guarantees the data is safely on disk when the completion signal comes in. The original article complains that UNIX/Linux file system semantics aren't well enough defined for database safety. He's right. They're close, but not quite there, because behavior after a crash is unspecified.

colanderman 11y ago
sync_file_range(2) allows you to specify ranges to sync, instead of the whole file, but it's not that useful anyway. In most DBs, you have separate data and log. The log is always only appended to, and the nature of disks is such that grouping concurrent transactions together into a single commit is a big win regardless of how syncing works. So syncing all outstanding I/O to the log file is generally what you want to do anyway. While the data store is always being written to, its writes do not need to be synced to disk until a log checkpoint is made (and the old log subsequently discarded). And since that is an intermittent asynchronous process, there is not much to gain from finer-grained syncing here either. See e.g. http://www.postgresql.org/message-id/4A51CB76.5020407@enterprisedb.com http://www.postgresql.org/message-id/4A51CB76.5020407@enterp...
- anarazel 11y ago
  Actually, we made postgres use sync_file_range() for checkpointing if available in 9.6. Not for durability - there's a few to many caveats in the manpage - but to control how much work a later fsync() has to do. In many workloads checkpointing can generate a lot of writes, and the OS's writeback caching of those can generate a lot of dirty buffers in the kernel's page cache. If the kernel decides to flush those (on it's own or due to an fsync), latency for every other FS operation can skyrocket. We've seen stalls in the 10s of minutes. So we now regularly sync_file_range(SYNC_FILE_RANGE_WRITE), unless the feature is disabled of course, to control how much dirty data the kernel has. See http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=428b1d6b29ca599c5700d4bc4f4ce4c5880369bf http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;... and http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=9cd00c457e6a1ebb984167ac556a9961812a683c http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;...

dspeyer 11y ago

Rename doesn't always do what we want. The one serious investigation I've seen [1] unhelpfully states that "[W]hen a file is appended, and then renamed ... many file systems recognize it and allocate blocks immediately." Suggesting that some file systems don't, but failing to name them. In general, appends followed by directory operations are not order-preserved. [1] https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14...

jstimpfle 11y ago
There's no order preserving guarantee. Code should do fd <- open(tmpfile) write(fd, data) fsync(fd) rename(tmpfile, finalpath) fsync(parent(finalpath)) (Strictly speaking there is a problem in the rename because we can't be sure "tmpfile" still refers to the same inode. It would be nice to have a way to link fds, but I'm not aware of any). According to the paper, existing file systems recognize the (open,) write, rename pattern and don't reorder write and rename. That would break too much existing code. But these are filesystem-specific design decisions. Btw. it's just the same with CPUs writing to memory. Generally writes can be reordered if the behaviour is not visible to the executing thread. However if there are multiple threads you might need to insert (CPU-specific) memory barriers to prevent reorderings that are visible to other threads. x86 is like those file systems here in that most incorrect code still works, because x86 makes some ordering guarantees.

caf 11y ago

rename(2) atomic replacement only means atomic with respect to processes running on the system at the same time: anyone looking up the newpath will get either the old file or the new file (and won't, for example, see a spurious ENOENT). It doesn't mean atomic with respect to a system crash. The easiest way to get asynchronous fsync(2) is to do the fsync(2) in a different thread.

jstimpfle 11y ago
> The easiest way to get asynchronous fsync(2) is to do the fsync(2) in a different thread. However you would need to create a new thread for each simultaneous asynchronous fsync to get the benefits of asynchronicity.