6 ms·

There are more wrinkles with this: - if you are creating a file, to ensure full synchronisation you also need to fsync the parent directory, otherwise the file

There are more wrinkles with this: - if you are creating a file, to ensure full synchronisation you also need to fsync the parent directory, otherwise the file can be fsynced but the update to the directory lost - if sync fails, you can not assume anything about the file, whether on-disk or in memory, critically one understanding which got dubbed "fsyncgate" and lead to many RDBMS having to be updated is that you can not portably retry fsync after failure: the earlier error may have invalidated the IO buffers but IO errors may not be sticky, so a later fsync will have nothing to write and report success

dist-epoch 2y ago
So if I do fopen/fwrite/fsync/fclose, that is not enough? That is crazy, I think 90% of apps don't fsync the parent directory. Also, how many levels of parents do you need to fsync?
- masklinn 2y ago
  > So if I do fopen/fwrite/fsync/fclose, that is not enough? That is my understanding. > Also, how many levels of parents do you need to fsync? Only one, at least if you didn't create the parent directory (if you did then you might have to fsync its parent, recursively). The fsync on the parent directory ensures the dir entry for your new file is flushed to disk.
  - magicalhippo 2y ago
    I've never heard about this in my years of programming. I just tried to read through the Win32 documentation, as I've done several times over the years, and it mentions a lot of edge cases but not this that I could see. Is this some Linux/Unix specific thing? Am I blind?
    - masklinn 2y ago
      I am talking about posix semantics yes, I have no idea how things work on windows.
      - magicalhippo 2y ago
        Phew. I've primarily used Windows. Not that any of the posix programs I've been exposed to have done the dir sync though. For cross-platform stuff I've mainly used Boost, which I assumed handled such details.
        guappa 2y ago
        I'm sure it does not. Also these things are needed very very rarely (which is why few even know about the issue) and are not good for performance and battery life.
        account42 2y ago
        More specifically, these things are for trying to improve the behavior on unclean system shutdown (e.g. power loss) which is inherently chaotic and unless all parts (most critically the disk and its controller) are well behaved you don't have any real guarantees anyway. Windows also doesn't guarantee that data is written to disk by the time WriteFile/CloseHandle returns, the Windows version of fsync is FlushFileBuffers.
      - red_admiral 2y ago
        There's the old saying that on Windows*, files have names; in POSIX, names have files. I think that's what makes the difference here. * technically it's the filesystem as much as the OS that is relevant here.
        pixl97 2y ago
        * a very limited set of names in relation to what's possible on other operating systems. If your processing files from other systems on NTFS you'll very likely have rename said files in an application and store an index of the names.
    - red_admiral 2y ago
      As far as I know, it's specific to the combination of certain POSIX-ish OS and file systems, like linux/Ext3. I have no clue what BSD does here, or whether ReiserFS is different. Windows/NTFS is a different world, there are still edge cases that can go wrong but I don't think this particular one is a problem because FAT/NTFS is not inode-based. I imagine if you looked at the SQLite source code you'd see different edge-case-handling code for different OSes.
      - pixl97 2y ago
        Ya NTFS file naming and long path is a wreck. Unzipping files from mac/linux is a easy way to end up with missing data. Applications quite often break on long file paths especially Microsofts own stuff like powershell.
      - cryptonector 2y ago
        NTFS has inodes. The thing about Windows is that because the file open operation (`CreateFile*()`) by default prevents renames of the file, Windows apps have come to not depend so much on file renaming, which makes one of the biggest filesystem power failure hazards less of an issue on Windows. But not being able to rename files over others as easily as in POSIX really sucks. And this doesn't completely absolve Windows app devs of having to think about power failure recovery! "POSIX semantics" is often short-hand for shortcomings of POSIX that happen to also be there in the WIN32 APIs, such as the lack of a filesystem write barrier, file stat-like system calls that mix different kinds of metadata (which sucks for distributed filesystems protocols), and so on. And yes, you can open files in Windows such that rename-over is allowed, so you still have this problem.
- jiggawatts 2y ago
  Crazy is the right term. File system APIs in general have too many sharp edges and need a ground-up rethink. Consider S3-like protocols: these recognise that 99% of the time applications just want “create file with given contents” or “read back what they’ve previously written.” The edge cases should be off the beaten path, not in your way tripping you up to when you want the simple scenario.
  - lionkor 2y ago
    It sounds more like you are asking for an abstraction
    - lmz 2y ago
      The file system is already an abstraction. I think they are asking if it's the right one.
  - Cthulhu_ 2y ago
    Aren't the edge cases features? An abstraction (or different API, sure) is in order to prevent footguns. However, this abstraction should not force fsyncs for example, due to the performance impact mentioned. It puts the choice of guaranteed writes vs performance to the developer.
    - aseipp 2y ago
      > Aren't the edge cases features? What features do you have in mind? > It puts the choice of guaranteed writes vs performance to the developer. Yes, and it's a completely false choice. This entire point of this thread is that fsync is an incredibly difficult API to use in a way that gets you the guarantees you need ("don't lose the writes to this file"). And that the consistency guarantees of specific filesystems, VFS, POSIX, and their interactions are not easy to understand even for the experienced -- and it can be catastrophic to get wrong. It isn't actually a choice between "Speed vs correctness". That's a nice fairy tale where people get to pretend they know what they're up against and everyone has full information. Most programmers aren't going to have the attention to get this right, even good ones. So then it's just "99.9% chance you fucked it up and it's wrong" and then your users are recovering data from backups.
- formerly_proven 2y ago
  One of the advantages of using a database for files is that it's relatively more likely that these platform-dependent considerations were indeed considered and you don't have to moonlight as an DBMS engineer while writing your application.
  - EasyMark 2y ago
    One reason Iike use sqlite so much in my personal projects, sometimes just using it to store various config data bits within row entries of the most generic table ever, works well with small projects, probably not so much with large scaled ones.
  - red_admiral 2y ago
    Hence the saying, "SQLite competes with fopen()".
- layer8 2y ago
  Yes. File system implementations never really thought this through, and hence here we are.
- everforward 2y ago
  That should be only for creating files, and maybe updating their metadata (not sure about that one). The confusion stems from people thinking that files and directories are more different than they are. Both are inodes, and both are basically containers for data. File inodes are containers for actual data, while directory inodes are containers for other inodes. All inodes need to be fsynced when you write to them. For files this is obviously when you write data to them. For directories, this is any time you change the inodes they contain, since you’re effectively writing data to them. You only need to sync the direct parent because the containers aren’t transitive; the grandparent directory only stores a reference to the parent directory, not to the files within the parent directory. It’s basically a graph.
  - xdavidliu 2y ago
    > It’s basically a graph. do you mean it's a basically a tree? Because if it were just a graph, you could still have edges from the grandparent to the grandchild in addition to the one from the former to the child
    - red_admiral 2y ago
      This ... depends. In normal POSIX land, hard links break the tree structure, for one, so you get a DAG but not a tree. I think some file systems do enforce tree structure, though - hard links are not supported everywhere. It used to be possible ages ago to hard link to directories, which meant that you could have actual cycles and a recursive tree-walking algorithm would never terminated. (As far as I know you can still do this by editing the disk manually, although I think fsck will make a fuss if it detects this.) You can still, with the right syscalls and drivers, do something like hard links on NTFS (I think they're technically called mount points but it's not the same thing as POSIX ones). I'm not sure if you can still make directory cycles, and you're probably a bad person if you do.
      - account42 2y ago
        Logically, ".." is a hard link to the parent directory. This may or may not actually be the case in the filesystem on the disk.
- anon-3988 2y ago
  Is anything ever enough aside from powering down the PC and then reading it back again to make sure its there?
- kccqzy 2y ago
  You don't need even to bother with fsync unless you are developing a database or use the file system like a database. There's a reason Apple made fsync useless and introduced F_FULLSYNC. They know developers have an incentive to overestimate the importance of their own files at the detriment to system responsiveness and power draw.
  - account42 2y ago
    Agreed, things like Firefox constantly fsyncing its profile database is absurd.
  - o11c 2y ago
    The problem is, Apple's fsync doesn't even introduce any useful semantics. All it says is "write this eventually". What a lot of people really need is just a "provide ordering, don't leave me with inconsistent data" operation. If you lose power "after" a write but before the fsync, for any non-networked application that's no different than losing power before the write, as long as the filesystem doesn't introduce chaos.
    - lxgr 2y ago
      That seems significantly harder to implement, though, given that dirty page cache entries don't have an order entry as far as I know, and so retroactively figuring out which writes not to reorder with others is anything but trivial.
- Verdex 2y ago
  Great another case of simple thing everyone knows is simple but turns out to be horrifyingly complicated and heavily system dependent, but only on occasion so you can get 80% of the way through your career before encountering the gaps in your knowledge. I guess I'll add it to the list. Of course on the other hand, I was already thinking that I should just use SQLite for all my file handling needs. This little nugget makes me think that this was the correct intuition. [Queue horrifying revelations w.r.t. SQLite here.]
  - garrettr_ 2y ago
    w.r.t SQLite, the only horrifying revelation I’ve had is that it allows NULLs in composite primary keys, which I’ve seen lead to some nasty bugs in practice.
    - cryptonector 2y ago
      Yes, you have to declare PRIMARY KEY columns as NOT NULL. There's lots of little caveats like this about SQLite3. So what.
- cryptonector 2y ago
  You need to `fflush()` too, before the `fsync()`.
praptak 2y ago
> otherwise the file can be fsynced but the update to the directory lost It also goes the other way - the update to the directory can be fsynced but the file lost. This can break the "create temp file, write, close, rename to current" scenario (when the intention is to replace file contents atomically). POSIX doesn't guarantee the order in which data hits the disk, so the above scenario can become "create temp file, write [contents still in memory only], rename to current [written to disk], power failure". I believe there was a bug where this scenario had worked for a long time then one filesystem (ext4?) pushed closer to what is admissible under POSIX (non-obvious reorders of physical writes) and people started getting random data corruption in programs which used write & rename.
- eatonphil 2y ago
  Can anyone provide links to learn more about these misconceptions and the bugs they caused? Edit: talking about filesystem misconceptions, not fsyncgate.
  - sgarland 2y ago
    EDIT: realized you were asking about the ext4 bugs, not the parent comment of that one. Oh well, keeping this up anyway for others. Postgres has a whole wiki page [0] about it, it's quite a read. They also link a [1] MySQL commit to fix the same issue. [0]: https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors [1]: https://github.com/mysql/mysql-server/commit/8590c8e12a3374eeccb547359750a9d2a128fa6a#diff-28ed40e7ccec6683ebd46da2ca82d01c https://github.com/mysql/mysql-server/commit/8590c8e12a3374e...
  - praptak 2y ago
    I have found a blog post from 2009 about this issue[1]. I think the "recent Ubuntu bug [that] has gotten slashdotted" is [2], so I think the most technical discussion about this would be [2]. I also found an interesting "scar tissue" from that bug in the current ext4 docs[0]: "If auto_da_alloc is enabled, ext4 will detect the replace-via-rename and replace-via-truncate patterns and force that any delayed allocation blocks are allocated such that at the next journal commit, in the default data=ordered mode, the data blocks of the new file are forced to disk before the rename() operation is committed. This provides roughly the same level of guarantees as ext3, and avoids the “zero-length” problem that can happen when a system crashes before the delayed allocation blocks are forced to disk." [0]https://docs.kernel.org/admin-guide/ext4.html https://docs.kernel.org/admin-guide/ext4.html [1] https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/ https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-a... [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781
weinzierl 2y ago
"if you are creating a file, to ensure full synchronisation you also need to fsync the parent directory, otherwise the file can be fsynced but the update to the directory lost." And if you need this in Java you still have resort to ugly hacks. https://github.com/apache/lucene/issues/7231 https://github.com/apache/lucene/issues/7231
- jryan49 2y ago
  This says the bug is fixed? https://bugs.openjdk.org/browse/JDK-8066915 https://bugs.openjdk.org/browse/JDK-8066915
  - weinzierl 2y ago
    The point is that there is no official way to fsync a directory in Java and that everyone is relying on an unintentional side effect of an unrelated function to accomplish it. The link I supplied is about the fact that the side effect briefly disappeared in Java 9 until enough people complained. We're still living in xkcd 1172 land with this, have been for a decade or who knows how long.
    - jryan49 2y ago
      Java does "abstract" the operating system away from you and in systems programming with java you can end up with leaky abstractions.
procflora 2y ago
More on fsyncgate: https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/
Brian_K_White 2y ago
"listen to what close() says" and "don't believe what close() says" are two different things. The article is only (initially) talking about the first, and that is valid. The second is just second-guessing the OS and hardware environment, and is invalid. Here's the rule to figure out if you need fsync() or not: "If you think you might need fsync(), you don't." ;) There are almost no cases where you should worry about the underlying layers. Basically if you aren't writing the filesystem itself, then you shouldn't be calling fsync(). You have to check what open()/close() etc said, but if close() said it worked, then it worked. You're done. The fact that lightning might have struck the drive just exactly then is not your problem and not something you should try to do anything about. Unreliable networks, busses, batteries, etc none of that changes this. Those things all already have their own layers with their own responsibilities to be doing all the necessary testing and verifying and retrying before they return a success code to you. There is open(...,O_SYNC) and mount -o sync (udev rules) for the case of a camera or thumb drive connect by usb etc. It's not merely that you don't have to, it's that it's actively wrong to. fsync() is just joggling someone else's elbow while they're trying to do their job, and if it seems to solve some problem, that actually just exposes that you have some logic or order of operations problem and you aren't doing your own job. "trust the other layer" or "trust the api contract" is unrelated to and does not conflict with "Be forgiving in your inputs and strict in your outputs.". It just means: Do: Check the returned error from, say, malloc(). Don't: Get a success from malloc() and then go try to do things to prove that malloc() actually did work. That would be insane and impossible because it would have to apply equally to everything, including every single keyword or function you would use as part of the verification. How do you know when you so much as set a value to variable that it actually got set? If you printf the variable to prove it, how do you know printf didn't lie? The logic for trusting close() is no different from the logic for trusting malloc(). Our responsibility is just to stay within the bounds of defined behavior and not make any assumptions about anything that isn't promised.
- lxgr 2y ago
  > The fact that lightning might have struck the drive just exactly then is not your problem [...] If you think you might need fsync(), you don't. [...] Those things all already have their own layers with their own responsibilities to be doing all the necessary testing [...] You're mixing up the concepts of durability and consistency in pretty significant ways here, and are implying that everybody that is fine with a lack of the former will also be fine with a lack of the latter. This is absolutely not true, and can cause extremely painful and hard-to-track-down bugs. For better or worse, there is no way to directly tell the OS "whatever you do, make sure you don't reorder these writes I just did with those other writes I'm about to do". The next best (portable) thing we have to achieve that outcome is fsync. It's a bit heavy handed, in that it gives you durability even if you only want consistency. That absolutely doesn't mean it's redundant, though. > Basically if you aren't writing the filesystem itself, then you shouldn't be calling fsync(). Given that fsync is a syscall, but file systems are generally implemented in the kernel, this is a pretty nonsensical statement by itself. File systems usually have (and need, for performance) a much lower level view of the underlying block storage, and fine-grained control over it. Just as one example, Linux has the concept of write barriers. Not using these correctly (in the filesystem driver) can cause data leaks across files owned by different users and processes.

dist-epoch 2y ago

So if I do fopen/fwrite/fsync/fclose, that is not enough? That is crazy, I think 90% of apps don't fsync the parent directory. Also, how many levels of parents do you need to fsync?

masklinn 2y ago
> So if I do fopen/fwrite/fsync/fclose, that is not enough? That is my understanding. > Also, how many levels of parents do you need to fsync? Only one, at least if you didn't create the parent directory (if you did then you might have to fsync its parent, recursively). The fsync on the parent directory ensures the dir entry for your new file is flushed to disk.
- magicalhippo 2y ago
  I've never heard about this in my years of programming. I just tried to read through the Win32 documentation, as I've done several times over the years, and it mentions a lot of edge cases but not this that I could see. Is this some Linux/Unix specific thing? Am I blind?
  - masklinn 2y ago
    I am talking about posix semantics yes, I have no idea how things work on windows.
    - magicalhippo 2y ago
      Phew. I've primarily used Windows. Not that any of the posix programs I've been exposed to have done the dir sync though. For cross-platform stuff I've mainly used Boost, which I assumed handled such details.
      - guappa 2y ago
        I'm sure it does not. Also these things are needed very very rarely (which is why few even know about the issue) and are not good for performance and battery life.
        account42 2y ago
        More specifically, these things are for trying to improve the behavior on unclean system shutdown (e.g. power loss) which is inherently chaotic and unless all parts (most critically the disk and its controller) are well behaved you don't have any real guarantees anyway. Windows also doesn't guarantee that data is written to disk by the time WriteFile/CloseHandle returns, the Windows version of fsync is FlushFileBuffers.
    - red_admiral 2y ago
      There's the old saying that on Windows*, files have names; in POSIX, names have files. I think that's what makes the difference here. * technically it's the filesystem as much as the OS that is relevant here.
      - pixl97 2y ago
        * a very limited set of names in relation to what's possible on other operating systems. If your processing files from other systems on NTFS you'll very likely have rename said files in an application and store an index of the names.
  - red_admiral 2y ago
    As far as I know, it's specific to the combination of certain POSIX-ish OS and file systems, like linux/Ext3. I have no clue what BSD does here, or whether ReiserFS is different. Windows/NTFS is a different world, there are still edge cases that can go wrong but I don't think this particular one is a problem because FAT/NTFS is not inode-based. I imagine if you looked at the SQLite source code you'd see different edge-case-handling code for different OSes.
    - pixl97 2y ago
      Ya NTFS file naming and long path is a wreck. Unzipping files from mac/linux is a easy way to end up with missing data. Applications quite often break on long file paths especially Microsofts own stuff like powershell.
    - cryptonector 2y ago
      NTFS has inodes. The thing about Windows is that because the file open operation (`CreateFile*()`) by default prevents renames of the file, Windows apps have come to not depend so much on file renaming, which makes one of the biggest filesystem power failure hazards less of an issue on Windows. But not being able to rename files over others as easily as in POSIX really sucks. And this doesn't completely absolve Windows app devs of having to think about power failure recovery! "POSIX semantics" is often short-hand for shortcomings of POSIX that happen to also be there in the WIN32 APIs, such as the lack of a filesystem write barrier, file stat-like system calls that mix different kinds of metadata (which sucks for distributed filesystems protocols), and so on. And yes, you can open files in Windows such that rename-over is allowed, so you still have this problem.
jiggawatts 2y ago
Crazy is the right term. File system APIs in general have too many sharp edges and need a ground-up rethink. Consider S3-like protocols: these recognise that 99% of the time applications just want “create file with given contents” or “read back what they’ve previously written.” The edge cases should be off the beaten path, not in your way tripping you up to when you want the simple scenario.
- lionkor 2y ago
  It sounds more like you are asking for an abstraction
  - lmz 2y ago
    The file system is already an abstraction. I think they are asking if it's the right one.
- Cthulhu_ 2y ago
  Aren't the edge cases features? An abstraction (or different API, sure) is in order to prevent footguns. However, this abstraction should not force fsyncs for example, due to the performance impact mentioned. It puts the choice of guaranteed writes vs performance to the developer.
  - aseipp 2y ago
    > Aren't the edge cases features? What features do you have in mind? > It puts the choice of guaranteed writes vs performance to the developer. Yes, and it's a completely false choice. This entire point of this thread is that fsync is an incredibly difficult API to use in a way that gets you the guarantees you need ("don't lose the writes to this file"). And that the consistency guarantees of specific filesystems, VFS, POSIX, and their interactions are not easy to understand even for the experienced -- and it can be catastrophic to get wrong. It isn't actually a choice between "Speed vs correctness". That's a nice fairy tale where people get to pretend they know what they're up against and everyone has full information. Most programmers aren't going to have the attention to get this right, even good ones. So then it's just "99.9% chance you fucked it up and it's wrong" and then your users are recovering data from backups.
formerly_proven 2y ago
One of the advantages of using a database for files is that it's relatively more likely that these platform-dependent considerations were indeed considered and you don't have to moonlight as an DBMS engineer while writing your application.
- EasyMark 2y ago
  One reason Iike use sqlite so much in my personal projects, sometimes just using it to store various config data bits within row entries of the most generic table ever, works well with small projects, probably not so much with large scaled ones.
- red_admiral 2y ago
  Hence the saying, "SQLite competes with fopen()".
layer8 2y ago
Yes. File system implementations never really thought this through, and hence here we are.
everforward 2y ago
That should be only for creating files, and maybe updating their metadata (not sure about that one). The confusion stems from people thinking that files and directories are more different than they are. Both are inodes, and both are basically containers for data. File inodes are containers for actual data, while directory inodes are containers for other inodes. All inodes need to be fsynced when you write to them. For files this is obviously when you write data to them. For directories, this is any time you change the inodes they contain, since you’re effectively writing data to them. You only need to sync the direct parent because the containers aren’t transitive; the grandparent directory only stores a reference to the parent directory, not to the files within the parent directory. It’s basically a graph.
- xdavidliu 2y ago
  > It’s basically a graph. do you mean it's a basically a tree? Because if it were just a graph, you could still have edges from the grandparent to the grandchild in addition to the one from the former to the child
  - red_admiral 2y ago
    This ... depends. In normal POSIX land, hard links break the tree structure, for one, so you get a DAG but not a tree. I think some file systems do enforce tree structure, though - hard links are not supported everywhere. It used to be possible ages ago to hard link to directories, which meant that you could have actual cycles and a recursive tree-walking algorithm would never terminated. (As far as I know you can still do this by editing the disk manually, although I think fsck will make a fuss if it detects this.) You can still, with the right syscalls and drivers, do something like hard links on NTFS (I think they're technically called mount points but it's not the same thing as POSIX ones). I'm not sure if you can still make directory cycles, and you're probably a bad person if you do.
    - account42 2y ago
      Logically, ".." is a hard link to the parent directory. This may or may not actually be the case in the filesystem on the disk.
anon-3988 2y ago
Is anything ever enough aside from powering down the PC and then reading it back again to make sure its there?
kccqzy 2y ago
You don't need even to bother with fsync unless you are developing a database or use the file system like a database. There's a reason Apple made fsync useless and introduced F_FULLSYNC. They know developers have an incentive to overestimate the importance of their own files at the detriment to system responsiveness and power draw.
- account42 2y ago
  Agreed, things like Firefox constantly fsyncing its profile database is absurd.
- o11c 2y ago
  The problem is, Apple's fsync doesn't even introduce any useful semantics. All it says is "write this eventually". What a lot of people really need is just a "provide ordering, don't leave me with inconsistent data" operation. If you lose power "after" a write but before the fsync, for any non-networked application that's no different than losing power before the write, as long as the filesystem doesn't introduce chaos.
  - lxgr 2y ago
    That seems significantly harder to implement, though, given that dirty page cache entries don't have an order entry as far as I know, and so retroactively figuring out which writes not to reorder with others is anything but trivial.
Verdex 2y ago
Great another case of simple thing everyone knows is simple but turns out to be horrifyingly complicated and heavily system dependent, but only on occasion so you can get 80% of the way through your career before encountering the gaps in your knowledge. I guess I'll add it to the list. Of course on the other hand, I was already thinking that I should just use SQLite for all my file handling needs. This little nugget makes me think that this was the correct intuition. [Queue horrifying revelations w.r.t. SQLite here.]
- garrettr_ 2y ago
  w.r.t SQLite, the only horrifying revelation I’ve had is that it allows NULLs in composite primary keys, which I’ve seen lead to some nasty bugs in practice.
  - cryptonector 2y ago
    Yes, you have to declare PRIMARY KEY columns as NOT NULL. There's lots of little caveats like this about SQLite3. So what.
cryptonector 2y ago
You need to `fflush()` too, before the `fsync()`.

praptak 2y ago

> otherwise the file can be fsynced but the update to the directory lost It also goes the other way - the update to the directory can be fsynced but the file lost. This can break the "create temp file, write, close, rename to current" scenario (when the intention is to replace file contents atomically). POSIX doesn't guarantee the order in which data hits the disk, so the above scenario can become "create temp file, write [contents still in memory only], rename to current [written to disk], power failure". I believe there was a bug where this scenario had worked for a long time then one filesystem (ext4?) pushed closer to what is admissible under POSIX (non-obvious reorders of physical writes) and people started getting random data corruption in programs which used write & rename.

eatonphil 2y ago
Can anyone provide links to learn more about these misconceptions and the bugs they caused? Edit: talking about filesystem misconceptions, not fsyncgate.
- sgarland 2y ago
  EDIT: realized you were asking about the ext4 bugs, not the parent comment of that one. Oh well, keeping this up anyway for others. Postgres has a whole wiki page [0] about it, it's quite a read. They also link a [1] MySQL commit to fix the same issue. [0]: https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors [1]: https://github.com/mysql/mysql-server/commit/8590c8e12a3374eeccb547359750a9d2a128fa6a#diff-28ed40e7ccec6683ebd46da2ca82d01c https://github.com/mysql/mysql-server/commit/8590c8e12a3374e...
- praptak 2y ago
  I have found a blog post from 2009 about this issue[1]. I think the "recent Ubuntu bug [that] has gotten slashdotted" is [2], so I think the most technical discussion about this would be [2]. I also found an interesting "scar tissue" from that bug in the current ext4 docs[0]: "If auto_da_alloc is enabled, ext4 will detect the replace-via-rename and replace-via-truncate patterns and force that any delayed allocation blocks are allocated such that at the next journal commit, in the default data=ordered mode, the data blocks of the new file are forced to disk before the rename() operation is committed. This provides roughly the same level of guarantees as ext3, and avoids the “zero-length” problem that can happen when a system crashes before the delayed allocation blocks are forced to disk." [0]https://docs.kernel.org/admin-guide/ext4.html https://docs.kernel.org/admin-guide/ext4.html [1] https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/ https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-a... [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781

weinzierl 2y ago

"if you are creating a file, to ensure full synchronisation you also need to fsync the parent directory, otherwise the file can be fsynced but the update to the directory lost." And if you need this in Java you still have resort to ugly hacks. https://github.com/apache/lucene/issues/7231 https://github.com/apache/lucene/issues/7231

jryan49 2y ago
This says the bug is fixed? https://bugs.openjdk.org/browse/JDK-8066915 https://bugs.openjdk.org/browse/JDK-8066915
- weinzierl 2y ago
  The point is that there is no official way to fsync a directory in Java and that everyone is relying on an unintentional side effect of an unrelated function to accomplish it. The link I supplied is about the fact that the side effect briefly disappeared in Java 9 until enough people complained. We're still living in xkcd 1172 land with this, have been for a decade or who knows how long.
  - jryan49 2y ago
    Java does "abstract" the operating system away from you and in systems programming with java you can end up with leaky abstractions.

procflora 2y ago

More on fsyncgate: https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/

Brian_K_White 2y ago

"listen to what close() says" and "don't believe what close() says" are two different things. The article is only (initially) talking about the first, and that is valid. The second is just second-guessing the OS and hardware environment, and is invalid. Here's the rule to figure out if you need fsync() or not: "If you think you might need fsync(), you don't." ;) There are almost no cases where you should worry about the underlying layers. Basically if you aren't writing the filesystem itself, then you shouldn't be calling fsync(). You have to check what open()/close() etc said, but if close() said it worked, then it worked. You're done. The fact that lightning might have struck the drive just exactly then is not your problem and not something you should try to do anything about. Unreliable networks, busses, batteries, etc none of that changes this. Those things all already have their own layers with their own responsibilities to be doing all the necessary testing and verifying and retrying before they return a success code to you. There is open(...,O_SYNC) and mount -o sync (udev rules) for the case of a camera or thumb drive connect by usb etc. It's not merely that you don't have to, it's that it's actively wrong to. fsync() is just joggling someone else's elbow while they're trying to do their job, and if it seems to solve some problem, that actually just exposes that you have some logic or order of operations problem and you aren't doing your own job. "trust the other layer" or "trust the api contract" is unrelated to and does not conflict with "Be forgiving in your inputs and strict in your outputs.". It just means: Do: Check the returned error from, say, malloc(). Don't: Get a success from malloc() and then go try to do things to prove that malloc() actually did work. That would be insane and impossible because it would have to apply equally to everything, including every single keyword or function you would use as part of the verification. How do you know when you so much as set a value to variable that it actually got set? If you printf the variable to prove it, how do you know printf didn't lie? The logic for trusting close() is no different from the logic for trusting malloc(). Our responsibility is just to stay within the bounds of defined behavior and not make any assumptions about anything that isn't promised.

lxgr 2y ago
> The fact that lightning might have struck the drive just exactly then is not your problem [...] If you think you might need fsync(), you don't. [...] Those things all already have their own layers with their own responsibilities to be doing all the necessary testing [...] You're mixing up the concepts of durability and consistency in pretty significant ways here, and are implying that everybody that is fine with a lack of the former will also be fine with a lack of the latter. This is absolutely not true, and can cause extremely painful and hard-to-track-down bugs. For better or worse, there is no way to directly tell the OS "whatever you do, make sure you don't reorder these writes I just did with those other writes I'm about to do". The next best (portable) thing we have to achieve that outcome is fsync. It's a bit heavy handed, in that it gives you durability even if you only want consistency. That absolutely doesn't mean it's redundant, though. > Basically if you aren't writing the filesystem itself, then you shouldn't be calling fsync(). Given that fsync is a syscall, but file systems are generally implemented in the kernel, this is a pretty nonsensical statement by itself. File systems usually have (and need, for performance) a much lower level view of the underlying block storage, and fine-grained control over it. Just as one example, Linux has the concept of write barriers. Not using these correctly (in the filesystem driver) can cause data leaks across files owned by different users and processes.