4 ms·

Which of these guarantee that data is committed to storage? flush() fsync() close() What about if the storage is on NFS? The reality is that, unl

Which of these guarantee that data is committed to storage? flush() fsync() close() What about if the storage is on NFS? The reality is that, unless you're heavily reliant on platform-specific behaviour and are certain you're not, for example, dealing with files over NFS, or hardware which happens to relax any guarantees you thought you had, nothing is really guaranteed. The best defense I've seen against the uncertainty of whether files are written is to use the strategy required when writing to a Maildir-format mailbox. This employs a directory used for temporary writes to unique filenames, followed by a move into place in a 'real' directory, where files should be accessible once write is deemed to be complete. Those who've been around for some time may have experienced corrupt mailboxes where the storage format was something mbox-alike - with all messages held in one file (and there's no fancy database-style transaction handling to deal with integrity). Any kind of cross-platform development where files are written to is made more difficult and less performant, thanks to this problem. In many cases, I'm afraid this is poorly understood, or ignored. I don't think there's even a way to measure how often data corruption occurs due to this not being accounted for.

generalizations 5y ago
> This employs a directory used for temporary writes to unique filenames, followed by a move into place in a 'real' directory, where files should be accessible once write is deemed to be complete. This still seems like it makes the assumption that the move will complete. Or is 'mv' significantly different from the full write?
- pkhuong 5y ago
  The move (rename) is a metadata-only operation and is atomic on POSIX-compliant platforms and in reality.
  - generalizations 5y ago
    Sure; IIRC, it only changes the inode. However, it's still trying to change bits on the disk, regardless if those are data or metadata. Can the metadata-only bits that the disk claims to write in order to implement the move be trusted to have been written?
  - steerablesafe 5y ago
    It might be atomic on the OS level (no processes can see an "intermediate" state), but it might not be atomic when you yank the power cord. POSIX in general doesn't offer any guarantees there, AFAIK.
    - deleted 5y ago
      [deleted]
  - yakubin 5y ago
    Rename being atomic doesn't imply that after a rename completes, it's flushed to cold storage. It just implies that a rename is either completely flushed to cold storage, or the data in cold storage looks as if there was never a rename. Also, the data you wrote to a file before the rename may not have been flushed completely to cold storage. Rename doesn't change here anything. The data may still be served from the disk controller's cache. The rename trick is mostly useful for "committing" long operations, which even on the high level are not atomic. But it has nothing to do with flushing. Also, renaming a file is a filesystem-level operation. From the POV of a disk, it looks the same as any other write. So if you can't make sure that normal writes are flushed, you can't make sure that renames are flushed. Another note: rename(2) used not to be even atomic wrt power loss on ext4 (which was later fixed), so it's not guaranteed. POSIX itself only guarantees atomicity wrt other filesystem operations. Atomicity against power loss is a distinct checkbox. You need to check your specific filesystem.
    - cout 5y ago
      I'll add this: on spinning disks it's normal to reorder write operations; it is optimal to order writes in a way that matches the layout on disk. That way, each sector is written as the write head passes over the sector. We like to think of a disk-backed filesystem as slow, persistent RAM, but in reality it's a lot more complex than that.
MobiusHorizons 5y ago
Any such guarantees are only true for the OS and filesystem (ie the data has been transferred from main memory across whatever physical bus the disk is attached to ide/scsi/sata/spi/network/nvme, etc) In modern drives, the specific write operation may be queued up on the physical disk in volatile storage outside the control of the operating system. A flush can’t help you with this problem, flush is for the operating system to flush its cache to disk. This is where F_FULLFSYNC comes in. It is a request to the disk to finish flushing the volatile caches to nonvolatile storage. Some disks just ignore this request, claiming to be done before that is actually true. In these cases no userland or even kernel level programming tricks can save you from data loss, since you can’t actually know when the data becomes safe on the disk.
reincarnate0x14 5y ago
> In many cases, I'm afraid this is poorly understood, or ignored. Definite agreement on that. In many ways it's even less understood nowadays as pervasive reliable building power and UPSes are largely taken for granted, so the operational knowledge of the problem has been fading away until some cloud service loses an important circuit and suddenly a database has unpredictable data corruption.

generalizations 5y ago

> This employs a directory used for temporary writes to unique filenames, followed by a move into place in a 'real' directory, where files should be accessible once write is deemed to be complete. This still seems like it makes the assumption that the move will complete. Or is 'mv' significantly different from the full write?

pkhuong 5y ago
The move (rename) is a metadata-only operation and is atomic on POSIX-compliant platforms and in reality.
- generalizations 5y ago
  Sure; IIRC, it only changes the inode. However, it's still trying to change bits on the disk, regardless if those are data or metadata. Can the metadata-only bits that the disk claims to write in order to implement the move be trusted to have been written?
- steerablesafe 5y ago
  It might be atomic on the OS level (no processes can see an "intermediate" state), but it might not be atomic when you yank the power cord. POSIX in general doesn't offer any guarantees there, AFAIK.
  - deleted 5y ago
    [deleted]
- yakubin 5y ago
  Rename being atomic doesn't imply that after a rename completes, it's flushed to cold storage. It just implies that a rename is either completely flushed to cold storage, or the data in cold storage looks as if there was never a rename. Also, the data you wrote to a file before the rename may not have been flushed completely to cold storage. Rename doesn't change here anything. The data may still be served from the disk controller's cache. The rename trick is mostly useful for "committing" long operations, which even on the high level are not atomic. But it has nothing to do with flushing. Also, renaming a file is a filesystem-level operation. From the POV of a disk, it looks the same as any other write. So if you can't make sure that normal writes are flushed, you can't make sure that renames are flushed. Another note: rename(2) used not to be even atomic wrt power loss on ext4 (which was later fixed), so it's not guaranteed. POSIX itself only guarantees atomicity wrt other filesystem operations. Atomicity against power loss is a distinct checkbox. You need to check your specific filesystem.
  - cout 5y ago
    I'll add this: on spinning disks it's normal to reorder write operations; it is optimal to order writes in a way that matches the layout on disk. That way, each sector is written as the write head passes over the sector. We like to think of a disk-backed filesystem as slow, persistent RAM, but in reality it's a lot more complex than that.

MobiusHorizons 5y ago

Any such guarantees are only true for the OS and filesystem (ie the data has been transferred from main memory across whatever physical bus the disk is attached to ide/scsi/sata/spi/network/nvme, etc) In modern drives, the specific write operation may be queued up on the physical disk in volatile storage outside the control of the operating system. A flush can’t help you with this problem, flush is for the operating system to flush its cache to disk. This is where F_FULLFSYNC comes in. It is a request to the disk to finish flushing the volatile caches to nonvolatile storage. Some disks just ignore this request, claiming to be done before that is actually true. In these cases no userland or even kernel level programming tricks can save you from data loss, since you can’t actually know when the data becomes safe on the disk.

reincarnate0x14 5y ago

> In many cases, I'm afraid this is poorly understood, or ignored. Definite agreement on that. In many ways it's even less understood nowadays as pervasive reliable building power and UPSes are largely taken for granted, so the operational knowledge of the problem has been fading away until some cloud service loses an important circuit and suddenly a database has unpredictable data corruption.