5 ms·

So if I do fopen/fwrite/fsync/fclose, that is not enough? That is crazy, I think 90% of apps don't fsync the parent directory. Also, how many levels of parents

So if I do fopen/fwrite/fsync/fclose, that is not enough? That is crazy, I think 90% of apps don't fsync the parent directory. Also, how many levels of parents do you need to fsync?

masklinn 2y ago
> So if I do fopen/fwrite/fsync/fclose, that is not enough? That is my understanding. > Also, how many levels of parents do you need to fsync? Only one, at least if you didn't create the parent directory (if you did then you might have to fsync its parent, recursively). The fsync on the parent directory ensures the dir entry for your new file is flushed to disk.
- magicalhippo 2y ago
  I've never heard about this in my years of programming. I just tried to read through the Win32 documentation, as I've done several times over the years, and it mentions a lot of edge cases but not this that I could see. Is this some Linux/Unix specific thing? Am I blind?
  - masklinn 2y ago
    I am talking about posix semantics yes, I have no idea how things work on windows.
    - magicalhippo 2y ago
      Phew. I've primarily used Windows. Not that any of the posix programs I've been exposed to have done the dir sync though. For cross-platform stuff I've mainly used Boost, which I assumed handled such details.
      - guappa 2y ago
        I'm sure it does not. Also these things are needed very very rarely (which is why few even know about the issue) and are not good for performance and battery life.
        account42 2y ago
        More specifically, these things are for trying to improve the behavior on unclean system shutdown (e.g. power loss) which is inherently chaotic and unless all parts (most critically the disk and its controller) are well behaved you don't have any real guarantees anyway. Windows also doesn't guarantee that data is written to disk by the time WriteFile/CloseHandle returns, the Windows version of fsync is FlushFileBuffers.
    - red_admiral 2y ago
      There's the old saying that on Windows*, files have names; in POSIX, names have files. I think that's what makes the difference here. * technically it's the filesystem as much as the OS that is relevant here.
      - pixl97 2y ago
        * a very limited set of names in relation to what's possible on other operating systems. If your processing files from other systems on NTFS you'll very likely have rename said files in an application and store an index of the names.
  - red_admiral 2y ago
    As far as I know, it's specific to the combination of certain POSIX-ish OS and file systems, like linux/Ext3. I have no clue what BSD does here, or whether ReiserFS is different. Windows/NTFS is a different world, there are still edge cases that can go wrong but I don't think this particular one is a problem because FAT/NTFS is not inode-based. I imagine if you looked at the SQLite source code you'd see different edge-case-handling code for different OSes.
    - pixl97 2y ago
      Ya NTFS file naming and long path is a wreck. Unzipping files from mac/linux is a easy way to end up with missing data. Applications quite often break on long file paths especially Microsofts own stuff like powershell.
    - cryptonector 2y ago
      NTFS has inodes. The thing about Windows is that because the file open operation (`CreateFile*()`) by default prevents renames of the file, Windows apps have come to not depend so much on file renaming, which makes one of the biggest filesystem power failure hazards less of an issue on Windows. But not being able to rename files over others as easily as in POSIX really sucks. And this doesn't completely absolve Windows app devs of having to think about power failure recovery! "POSIX semantics" is often short-hand for shortcomings of POSIX that happen to also be there in the WIN32 APIs, such as the lack of a filesystem write barrier, file stat-like system calls that mix different kinds of metadata (which sucks for distributed filesystems protocols), and so on. And yes, you can open files in Windows such that rename-over is allowed, so you still have this problem.
jiggawatts 2y ago
Crazy is the right term. File system APIs in general have too many sharp edges and need a ground-up rethink. Consider S3-like protocols: these recognise that 99% of the time applications just want “create file with given contents” or “read back what they’ve previously written.” The edge cases should be off the beaten path, not in your way tripping you up to when you want the simple scenario.
- lionkor 2y ago
  It sounds more like you are asking for an abstraction
  - lmz 2y ago
    The file system is already an abstraction. I think they are asking if it's the right one.
- Cthulhu_ 2y ago
  Aren't the edge cases features? An abstraction (or different API, sure) is in order to prevent footguns. However, this abstraction should not force fsyncs for example, due to the performance impact mentioned. It puts the choice of guaranteed writes vs performance to the developer.
  - aseipp 2y ago
    > Aren't the edge cases features? What features do you have in mind? > It puts the choice of guaranteed writes vs performance to the developer. Yes, and it's a completely false choice. This entire point of this thread is that fsync is an incredibly difficult API to use in a way that gets you the guarantees you need ("don't lose the writes to this file"). And that the consistency guarantees of specific filesystems, VFS, POSIX, and their interactions are not easy to understand even for the experienced -- and it can be catastrophic to get wrong. It isn't actually a choice between "Speed vs correctness". That's a nice fairy tale where people get to pretend they know what they're up against and everyone has full information. Most programmers aren't going to have the attention to get this right, even good ones. So then it's just "99.9% chance you fucked it up and it's wrong" and then your users are recovering data from backups.
formerly_proven 2y ago
One of the advantages of using a database for files is that it's relatively more likely that these platform-dependent considerations were indeed considered and you don't have to moonlight as an DBMS engineer while writing your application.
- EasyMark 2y ago
  One reason Iike use sqlite so much in my personal projects, sometimes just using it to store various config data bits within row entries of the most generic table ever, works well with small projects, probably not so much with large scaled ones.
- red_admiral 2y ago
  Hence the saying, "SQLite competes with fopen()".
layer8 2y ago
Yes. File system implementations never really thought this through, and hence here we are.
everforward 2y ago
That should be only for creating files, and maybe updating their metadata (not sure about that one). The confusion stems from people thinking that files and directories are more different than they are. Both are inodes, and both are basically containers for data. File inodes are containers for actual data, while directory inodes are containers for other inodes. All inodes need to be fsynced when you write to them. For files this is obviously when you write data to them. For directories, this is any time you change the inodes they contain, since you’re effectively writing data to them. You only need to sync the direct parent because the containers aren’t transitive; the grandparent directory only stores a reference to the parent directory, not to the files within the parent directory. It’s basically a graph.
- xdavidliu 2y ago
  > It’s basically a graph. do you mean it's a basically a tree? Because if it were just a graph, you could still have edges from the grandparent to the grandchild in addition to the one from the former to the child
  - red_admiral 2y ago
    This ... depends. In normal POSIX land, hard links break the tree structure, for one, so you get a DAG but not a tree. I think some file systems do enforce tree structure, though - hard links are not supported everywhere. It used to be possible ages ago to hard link to directories, which meant that you could have actual cycles and a recursive tree-walking algorithm would never terminated. (As far as I know you can still do this by editing the disk manually, although I think fsck will make a fuss if it detects this.) You can still, with the right syscalls and drivers, do something like hard links on NTFS (I think they're technically called mount points but it's not the same thing as POSIX ones). I'm not sure if you can still make directory cycles, and you're probably a bad person if you do.
    - account42 2y ago
      Logically, ".." is a hard link to the parent directory. This may or may not actually be the case in the filesystem on the disk.
anon-3988 2y ago
Is anything ever enough aside from powering down the PC and then reading it back again to make sure its there?
kccqzy 2y ago
You don't need even to bother with fsync unless you are developing a database or use the file system like a database. There's a reason Apple made fsync useless and introduced F_FULLSYNC. They know developers have an incentive to overestimate the importance of their own files at the detriment to system responsiveness and power draw.
- account42 2y ago
  Agreed, things like Firefox constantly fsyncing its profile database is absurd.
- o11c 2y ago
  The problem is, Apple's fsync doesn't even introduce any useful semantics. All it says is "write this eventually". What a lot of people really need is just a "provide ordering, don't leave me with inconsistent data" operation. If you lose power "after" a write but before the fsync, for any non-networked application that's no different than losing power before the write, as long as the filesystem doesn't introduce chaos.
  - lxgr 2y ago
    That seems significantly harder to implement, though, given that dirty page cache entries don't have an order entry as far as I know, and so retroactively figuring out which writes not to reorder with others is anything but trivial.
Verdex 2y ago
Great another case of simple thing everyone knows is simple but turns out to be horrifyingly complicated and heavily system dependent, but only on occasion so you can get 80% of the way through your career before encountering the gaps in your knowledge. I guess I'll add it to the list. Of course on the other hand, I was already thinking that I should just use SQLite for all my file handling needs. This little nugget makes me think that this was the correct intuition. [Queue horrifying revelations w.r.t. SQLite here.]
- garrettr_ 2y ago
  w.r.t SQLite, the only horrifying revelation I’ve had is that it allows NULLs in composite primary keys, which I’ve seen lead to some nasty bugs in practice.
  - cryptonector 2y ago
    Yes, you have to declare PRIMARY KEY columns as NOT NULL. There's lots of little caveats like this about SQLite3. So what.
cryptonector 2y ago
You need to `fflush()` too, before the `fsync()`.

So if I do fopen/fwrite/fsync/fclose, that is not enough? That is crazy, I think 90% of apps don't fsync the parent directory. Also, how many levels of parents do you need to fsync?

masklinn 2y ago

> So if I do fopen/fwrite/fsync/fclose, that is not enough? That is my understanding. > Also, how many levels of parents do you need to fsync? Only one, at least if you didn't create the parent directory (if you did then you might have to fsync its parent, recursively). The fsync on the parent directory ensures the dir entry for your new file is flushed to disk.

magicalhippo 2y ago
I've never heard about this in my years of programming. I just tried to read through the Win32 documentation, as I've done several times over the years, and it mentions a lot of edge cases but not this that I could see. Is this some Linux/Unix specific thing? Am I blind?
- masklinn 2y ago
  I am talking about posix semantics yes, I have no idea how things work on windows.
  - magicalhippo 2y ago
    Phew. I've primarily used Windows. Not that any of the posix programs I've been exposed to have done the dir sync though. For cross-platform stuff I've mainly used Boost, which I assumed handled such details.
    - guappa 2y ago
      I'm sure it does not. Also these things are needed very very rarely (which is why few even know about the issue) and are not good for performance and battery life.
      - account42 2y ago
        More specifically, these things are for trying to improve the behavior on unclean system shutdown (e.g. power loss) which is inherently chaotic and unless all parts (most critically the disk and its controller) are well behaved you don't have any real guarantees anyway. Windows also doesn't guarantee that data is written to disk by the time WriteFile/CloseHandle returns, the Windows version of fsync is FlushFileBuffers.
  - red_admiral 2y ago
    There's the old saying that on Windows*, files have names; in POSIX, names have files. I think that's what makes the difference here. * technically it's the filesystem as much as the OS that is relevant here.
    - pixl97 2y ago
      * a very limited set of names in relation to what's possible on other operating systems. If your processing files from other systems on NTFS you'll very likely have rename said files in an application and store an index of the names.
- red_admiral 2y ago
  As far as I know, it's specific to the combination of certain POSIX-ish OS and file systems, like linux/Ext3. I have no clue what BSD does here, or whether ReiserFS is different. Windows/NTFS is a different world, there are still edge cases that can go wrong but I don't think this particular one is a problem because FAT/NTFS is not inode-based. I imagine if you looked at the SQLite source code you'd see different edge-case-handling code for different OSes.
  - pixl97 2y ago
    Ya NTFS file naming and long path is a wreck. Unzipping files from mac/linux is a easy way to end up with missing data. Applications quite often break on long file paths especially Microsofts own stuff like powershell.
  - cryptonector 2y ago
    NTFS has inodes. The thing about Windows is that because the file open operation (`CreateFile*()`) by default prevents renames of the file, Windows apps have come to not depend so much on file renaming, which makes one of the biggest filesystem power failure hazards less of an issue on Windows. But not being able to rename files over others as easily as in POSIX really sucks. And this doesn't completely absolve Windows app devs of having to think about power failure recovery! "POSIX semantics" is often short-hand for shortcomings of POSIX that happen to also be there in the WIN32 APIs, such as the lack of a filesystem write barrier, file stat-like system calls that mix different kinds of metadata (which sucks for distributed filesystems protocols), and so on. And yes, you can open files in Windows such that rename-over is allowed, so you still have this problem.

jiggawatts 2y ago

Crazy is the right term. File system APIs in general have too many sharp edges and need a ground-up rethink. Consider S3-like protocols: these recognise that 99% of the time applications just want “create file with given contents” or “read back what they’ve previously written.” The edge cases should be off the beaten path, not in your way tripping you up to when you want the simple scenario.

lionkor 2y ago
It sounds more like you are asking for an abstraction
- lmz 2y ago
  The file system is already an abstraction. I think they are asking if it's the right one.
Cthulhu_ 2y ago
Aren't the edge cases features? An abstraction (or different API, sure) is in order to prevent footguns. However, this abstraction should not force fsyncs for example, due to the performance impact mentioned. It puts the choice of guaranteed writes vs performance to the developer.
- aseipp 2y ago
  > Aren't the edge cases features? What features do you have in mind? > It puts the choice of guaranteed writes vs performance to the developer. Yes, and it's a completely false choice. This entire point of this thread is that fsync is an incredibly difficult API to use in a way that gets you the guarantees you need ("don't lose the writes to this file"). And that the consistency guarantees of specific filesystems, VFS, POSIX, and their interactions are not easy to understand even for the experienced -- and it can be catastrophic to get wrong. It isn't actually a choice between "Speed vs correctness". That's a nice fairy tale where people get to pretend they know what they're up against and everyone has full information. Most programmers aren't going to have the attention to get this right, even good ones. So then it's just "99.9% chance you fucked it up and it's wrong" and then your users are recovering data from backups.

formerly_proven 2y ago

One of the advantages of using a database for files is that it's relatively more likely that these platform-dependent considerations were indeed considered and you don't have to moonlight as an DBMS engineer while writing your application.

EasyMark 2y ago
One reason Iike use sqlite so much in my personal projects, sometimes just using it to store various config data bits within row entries of the most generic table ever, works well with small projects, probably not so much with large scaled ones.
red_admiral 2y ago
Hence the saying, "SQLite competes with fopen()".

layer8 2y ago

Yes. File system implementations never really thought this through, and hence here we are.

everforward 2y ago

That should be only for creating files, and maybe updating their metadata (not sure about that one). The confusion stems from people thinking that files and directories are more different than they are. Both are inodes, and both are basically containers for data. File inodes are containers for actual data, while directory inodes are containers for other inodes. All inodes need to be fsynced when you write to them. For files this is obviously when you write data to them. For directories, this is any time you change the inodes they contain, since you’re effectively writing data to them. You only need to sync the direct parent because the containers aren’t transitive; the grandparent directory only stores a reference to the parent directory, not to the files within the parent directory. It’s basically a graph.

xdavidliu 2y ago
> It’s basically a graph. do you mean it's a basically a tree? Because if it were just a graph, you could still have edges from the grandparent to the grandchild in addition to the one from the former to the child
- red_admiral 2y ago
  This ... depends. In normal POSIX land, hard links break the tree structure, for one, so you get a DAG but not a tree. I think some file systems do enforce tree structure, though - hard links are not supported everywhere. It used to be possible ages ago to hard link to directories, which meant that you could have actual cycles and a recursive tree-walking algorithm would never terminated. (As far as I know you can still do this by editing the disk manually, although I think fsck will make a fuss if it detects this.) You can still, with the right syscalls and drivers, do something like hard links on NTFS (I think they're technically called mount points but it's not the same thing as POSIX ones). I'm not sure if you can still make directory cycles, and you're probably a bad person if you do.
  - account42 2y ago
    Logically, ".." is a hard link to the parent directory. This may or may not actually be the case in the filesystem on the disk.

anon-3988 2y ago

Is anything ever enough aside from powering down the PC and then reading it back again to make sure its there?

kccqzy 2y ago

You don't need even to bother with fsync unless you are developing a database or use the file system like a database. There's a reason Apple made fsync useless and introduced F_FULLSYNC. They know developers have an incentive to overestimate the importance of their own files at the detriment to system responsiveness and power draw.

account42 2y ago
Agreed, things like Firefox constantly fsyncing its profile database is absurd.
o11c 2y ago
The problem is, Apple's fsync doesn't even introduce any useful semantics. All it says is "write this eventually". What a lot of people really need is just a "provide ordering, don't leave me with inconsistent data" operation. If you lose power "after" a write but before the fsync, for any non-networked application that's no different than losing power before the write, as long as the filesystem doesn't introduce chaos.
- lxgr 2y ago
  That seems significantly harder to implement, though, given that dirty page cache entries don't have an order entry as far as I know, and so retroactively figuring out which writes not to reorder with others is anything but trivial.

Verdex 2y ago

Great another case of simple thing everyone knows is simple but turns out to be horrifyingly complicated and heavily system dependent, but only on occasion so you can get 80% of the way through your career before encountering the gaps in your knowledge. I guess I'll add it to the list. Of course on the other hand, I was already thinking that I should just use SQLite for all my file handling needs. This little nugget makes me think that this was the correct intuition. [Queue horrifying revelations w.r.t. SQLite here.]

garrettr_ 2y ago
w.r.t SQLite, the only horrifying revelation I’ve had is that it allows NULLs in composite primary keys, which I’ve seen lead to some nasty bugs in practice.
- cryptonector 2y ago
  Yes, you have to declare PRIMARY KEY columns as NOT NULL. There's lots of little caveats like this about SQLite3. So what.

cryptonector 2y ago

You need to `fflush()` too, before the `fsync()`.