4 ms·

> otherwise the file can be fsynced but the update to the directory lost It also goes the other way - the update to the directory can be fsynced but the file l

> otherwise the file can be fsynced but the update to the directory lost It also goes the other way - the update to the directory can be fsynced but the file lost. This can break the "create temp file, write, close, rename to current" scenario (when the intention is to replace file contents atomically). POSIX doesn't guarantee the order in which data hits the disk, so the above scenario can become "create temp file, write [contents still in memory only], rename to current [written to disk], power failure". I believe there was a bug where this scenario had worked for a long time then one filesystem (ext4?) pushed closer to what is admissible under POSIX (non-obvious reorders of physical writes) and people started getting random data corruption in programs which used write & rename.

eatonphil 2y ago
Can anyone provide links to learn more about these misconceptions and the bugs they caused? Edit: talking about filesystem misconceptions, not fsyncgate.
- sgarland 2y ago
  EDIT: realized you were asking about the ext4 bugs, not the parent comment of that one. Oh well, keeping this up anyway for others. Postgres has a whole wiki page [0] about it, it's quite a read. They also link a [1] MySQL commit to fix the same issue. [0]: https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors [1]: https://github.com/mysql/mysql-server/commit/8590c8e12a3374eeccb547359750a9d2a128fa6a#diff-28ed40e7ccec6683ebd46da2ca82d01c https://github.com/mysql/mysql-server/commit/8590c8e12a3374e...
- praptak 2y ago
  I have found a blog post from 2009 about this issue[1]. I think the "recent Ubuntu bug [that] has gotten slashdotted" is [2], so I think the most technical discussion about this would be [2]. I also found an interesting "scar tissue" from that bug in the current ext4 docs[0]: "If auto_da_alloc is enabled, ext4 will detect the replace-via-rename and replace-via-truncate patterns and force that any delayed allocation blocks are allocated such that at the next journal commit, in the default data=ordered mode, the data blocks of the new file are forced to disk before the rename() operation is committed. This provides roughly the same level of guarantees as ext3, and avoids the “zero-length” problem that can happen when a system crashes before the delayed allocation blocks are forced to disk." [0]https://docs.kernel.org/admin-guide/ext4.html https://docs.kernel.org/admin-guide/ext4.html [1] https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/ https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-a... [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781
deleted 2y ago
[deleted]
nextaccountic 2y ago
> It also goes the other way - the update to the directory can be fsynced but the file lost. This can break the "create temp file, write, close, rename to current" scenario (when the intention is to replace file contents atomically). > POSIX doesn't guarantee the order in which data hits the disk, so the above scenario can become "create temp file, write [contents still in memory only], rename to current [written to disk], power failure". Wait.. is there a way to do this correctly? At which points is fsync warranted, and how many fsyncs do we need for the whole "write file then mv it on top of current" to not lose data on power failure?
- paulsutter 2y ago
  Sqlite is much better than raw files to keep data intact on power failures, and be sure to study the options carefully If you truly need to use files you can take other steps such as mv the old file to .bck before mv the new file, but I really think you want sqlite
  - account42 2y ago
    SQLite doesn't do any magic other than fsync. Using it to deal with power failures is nonsense.
    - vlovich123 2y ago
      Well generally OP is correct that SQLite will more likely have written fsync code correctly and be fairly robust against power failures due to the WAL. Additionally, it’s not doing any renaming which you would need with plain files. So while it doesn’t do any other magic, it’s more likely to handle power failures correctly.
    - cryptonector 2y ago
      `fsync()` is not necessarily enough if your on-disk format is complex because you still need to write recovery code. Of course, SQLite3's on-disk format is very complex, and so it requires correspondingly more complex power failure recovery code. But SQLite3 has an excellent test suite, and in particular they have an excellent power failure recovery test suite. Using SQLite3 to avoid power failure issues is pretty good advice. I don't see why GP is getting downvoted.

eatonphil 2y ago

Can anyone provide links to learn more about these misconceptions and the bugs they caused? Edit: talking about filesystem misconceptions, not fsyncgate.

sgarland 2y ago
EDIT: realized you were asking about the ext4 bugs, not the parent comment of that one. Oh well, keeping this up anyway for others. Postgres has a whole wiki page [0] about it, it's quite a read. They also link a [1] MySQL commit to fix the same issue. [0]: https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors [1]: https://github.com/mysql/mysql-server/commit/8590c8e12a3374eeccb547359750a9d2a128fa6a#diff-28ed40e7ccec6683ebd46da2ca82d01c https://github.com/mysql/mysql-server/commit/8590c8e12a3374e...
praptak 2y ago
I have found a blog post from 2009 about this issue[1]. I think the "recent Ubuntu bug [that] has gotten slashdotted" is [2], so I think the most technical discussion about this would be [2]. I also found an interesting "scar tissue" from that bug in the current ext4 docs[0]: "If auto_da_alloc is enabled, ext4 will detect the replace-via-rename and replace-via-truncate patterns and force that any delayed allocation blocks are allocated such that at the next journal commit, in the default data=ordered mode, the data blocks of the new file are forced to disk before the rename() operation is committed. This provides roughly the same level of guarantees as ext3, and avoids the “zero-length” problem that can happen when a system crashes before the delayed allocation blocks are forced to disk." [0]https://docs.kernel.org/admin-guide/ext4.html https://docs.kernel.org/admin-guide/ext4.html [1] https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/ https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-a... [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781

deleted 2y ago

[deleted]

nextaccountic 2y ago

> It also goes the other way - the update to the directory can be fsynced but the file lost. This can break the "create temp file, write, close, rename to current" scenario (when the intention is to replace file contents atomically). > POSIX doesn't guarantee the order in which data hits the disk, so the above scenario can become "create temp file, write [contents still in memory only], rename to current [written to disk], power failure". Wait.. is there a way to do this correctly? At which points is fsync warranted, and how many fsyncs do we need for the whole "write file then mv it on top of current" to not lose data on power failure?

paulsutter 2y ago
Sqlite is much better than raw files to keep data intact on power failures, and be sure to study the options carefully If you truly need to use files you can take other steps such as mv the old file to .bck before mv the new file, but I really think you want sqlite
- account42 2y ago
  SQLite doesn't do any magic other than fsync. Using it to deal with power failures is nonsense.
  - vlovich123 2y ago
    Well generally OP is correct that SQLite will more likely have written fsync code correctly and be fairly robust against power failures due to the WAL. Additionally, it’s not doing any renaming which you would need with plain files. So while it doesn’t do any other magic, it’s more likely to handle power failures correctly.
  - cryptonector 2y ago
    `fsync()` is not necessarily enough if your on-disk format is complex because you still need to write recovery code. Of course, SQLite3's on-disk format is very complex, and so it requires correspondingly more complex power failure recovery code. But SQLite3 has an excellent test suite, and in particular they have an excellent power failure recovery test suite. Using SQLite3 to avoid power failure issues is pretty good advice. I don't see why GP is getting downvoted.