6 ms·

In Apple defense, sloppy fsync behaviour is clearly documented: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/ma

In Apple defense, sloppy fsync behaviour is clearly documented: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fsync.2.html https://developer.apple.com/library/archive/documentation/Sy...

lbriner 5y ago
That's not defence. It fails the principle of least-surprise. If everyone's experience is that fsync is flushing then why would somebody think to look up the docs for Mac in case they do it differently?
- supermatt 5y ago
  They do it according to POSIX spec. Linux is the oddball here.
  - marcan_42 5y ago
    The POSIX spec is deliberately ambiguous about this.
    - coldtea 5y ago
      So? Did Linux do it like that before? https://lwn.net/Articles/270891/ https://lwn.net/Articles/270891/
      - marcan_42 5y ago
        It does it like that now, which is what I'd expect if I'm writing software.
        wil421 5y ago
        I’d argue more people develop on non-Linux devices such as Windows and MacOS on linux itself.
        marcan_42 5y ago
        F_FULLFSYNC is nonstandard. As far as I know there is no standard-complicant way to get data on to stable storage on macOS. That's a bit of a problem. It makes a lot more sense to make the standard-compliant way actually sane.
        supermatt 5y ago
        I have said a few times already - F_BARRIERFSYNC. This is likely equivalent to what linux is doing. edit: sorry - not 'standards compliant' (whatever that is - does linux declare support for SIO?), but probably what you are looking for.
        marcan_42 5y ago
        It isn't. I already replied to you above. A barrier does not guarantee data durability and we already know Linux fsync() == macOS F_FULLFSYNC because they have the same (lack of) performance on the same hardware.
        supermatt 5y ago
        Thanks marcan. Apologies for wasting your time.
        deleted 5y ago
        [deleted]
        ryao 5y ago
        I just looked into this, since what you say and what Apple’s documentation says are two different things. Here is Apple’s documentation: https://devstreaming-cdn.apple.com/videos/wwdc/2019/419ef9ipua3hnd2y/419/419_optimizing_storage_in_your_app.pdf?dl=1 https://devstreaming-cdn.apple.com/videos/wwdc/2019/419ef9ip... F_BARRIERFSYNC: fsync() with a barrier F_FULLFSYNC: Drive flush its cache to disk This sounds like the Linux fsync() and Linux syncfs() respectively. What you say is that F_FULLFSYNC is the same as Linux fsync() and your performance numbers back that up. Unfortunately, you would only see a difference between Linux fsync() and Linux syncfs() if you have files being asynchronously written at the same time as the files that are subject to fsync()/syncfs(). fsync() would only touch the chosen files while syncfs() would touch both. If you did not have heavy background file writes and F_FULLSYNC really is equivalent to syncfs(), you would not be able to tell the difference in your tests. That said, let’s look at how this actually works on Mac OS. Unfortunately, the apfs driver does not appear to be open source, but the HFS+ driver is. Here are the relevant pieces of code in HFS+: https://github.com/apple-oss-distributions/hfs/blob/hfs-556.100.11/core/hfs_vnops.c#L3230 https://github.com/apple-oss-distributions/hfs/blob/hfs-556.... https://github.com/apple-oss-distributions/hfs/blob/5e3008b69b8e4e893fb240d3a2f09dbdf7d3c0fb/core/hfs_vfsutils.c#L3419 https://github.com/apple-oss-distributions/hfs/blob/5e3008b6... First, let me start with saying this merits a faceplam. The fsync() operation is operating at the level of the mount point, not the individual file. F_FULLSYNC and F_BARRIERFSYNC are different, but they both might as well be variants of the Linux syncfs(). For good measure, let us look at how this is done on the MacOS ZFS driver: https://github.com/openzfsonosx/zfs/blob/master/module/zfs/zfs_vnops_osx.c#L451 https://github.com/openzfsonosx/zfs/blob/master/module/zfs/z... The file is properly synced independently of the mountpoint, such that other files being modified in the file are not immediately required to be written out to disk. That said, both F_FULLSYNC and F_BARRIERFSYNC on MacOS are mapped by the ZFS driver to the same function that implements fsync() on Linux: https://github.com/openzfs/zfs/blob/master/module/os/linux/zfs/zpl_file.c#L127 https://github.com/openzfs/zfs/blob/master/module/os/linux/z... For good measure, let us look at how syncfs() is implemented by ZFS on Linux: https://github.com/openzfs/zfs/blob/master/module/os/linux/zfs/zpl_super.c#L118 https://github.com/openzfs/zfs/blob/master/module/os/linux/z... It operates on the superblock, which is what MacOS’ HFS+ driver does. From this, I can conclude: Linux syncfs() == macOS F_FULLFSYNC on HFS+ Linux fsync() == macOS fsync()/F_FULLFSYNC/F_BARRIERFSYNC on ZFS Also, MacOS F_BARRIERSYNC is a weakened Linux syncfs() and Apple’s documentation is very misleading (although maybe not technically wrong). POSIX does allow fsync to be implemented via syncfs (sync in POSIX, but I am saying syncfs from Linux to be less confusing). However, not issuing and waiting for the completion of an IO barrier on fsync is broken behavior like you claim. I am not sure how MacOS APFS behaves. I imagine that additional testing that takes into account the nuances in semantics would be able to clarify that. If it behaves like HFS+, it is broken. Edit: Upon further examination and comparing notes with the MacOS ZFS driver team lead, it seems that HFS+ is syncing more than requested when F_FULLFSYNC is used, but less than the entire filesystem. You are fine treating it as a Linux fsync. It is close enough.
- olliej 5y ago
  Linux only stopped doing the clearly wrong thing in 2008 or so iirc. It is still dumb that there's a definition of fsync() that does not sync :-/
- threeseed 5y ago
  > why would somebody think to look up the docs It seems reckless to me to not do this when you're interacting with the filesystem using low-level APIs (i.e not via Swift/Obj-C).
- masklinn 5y ago
  > That's not defence. It fails the principle of least-surprise. Welcome to C APIs in general, and POSIX in particular.
- coldtea 5y ago
  >That's not defence. It fails the principle of least-surprise. Only if the standard where anything else is a "surprise" is 2022 Linux. Many (all?) other unices and macOS itself since forever work like that. Including Linux itself in the past [1] [1] https://lwn.net/Articles/270891/ https://lwn.net/Articles/270891/
  - marcan_42 5y ago
    Drive caches also used to not exist in the past. At that point, behavior was the same as it is on Linux today. It then regressed when drive caches became a thing. Maybe it not being added to OSes when drive caches came into the picture was arguably a bug, and Linux has been the first OS to fix it properly. macOS instead introduced new, non-buggy behavior, and left the buggy one behind :-)
    - acdha 5y ago
      > Drive caches also used to not exist in the past. At that point, behavior was the same as it is on Linux today. It then regressed when drive caches became a thing. You mean in the 1980s? Linux wasn’t used before this wasn’t a concern for sysadmins and DBAs. This concern has been raised for years - back in the PowerPC era the numbers were lower but you had the same arguments about whether Apple had made the right trade-offs, or Linux or Solaris, etc. Given the extreme rarity of filesystem corruption being a problem these days, one might conclude that the engineers who made the assumption that batteries covered laptop users and anyone who cares about this will be using clustering / UPS were correct.
    - CyberRabbi 5y ago
      The minute storage manufacturers introduced drive caches is the minute this bug became the responsibility of storage manufacturers. IMO it’s not the kernel’s responsibility.
      - fomine3 5y ago
        Now Apple is the primary storage manufacturer for Mac.
- ecf 5y ago
  I’d argue maybe .5% of people are working on something where this is even close to being a concern. Those people probably know what they need to use. Apple doesn’t need to defend anything.
  - ClumsyPilot 5y ago
    I am sick of this callous and capricious disrespect for users and their data, rampant throughtout this wanky industry. Do lawyers use Apple computers? Do they work on important documents relating to life and death? Some people have literally been executed because developers couldn't do their job properly. People have been sent to jail for decades because developers fucked up in the british postmaster scandal. Average people life in a dangerous world- work with documents about their financial wellbeing. They live in opressive countries where being gay is punishable by death. They drive 2 ton death machines. And now that we have put computers in places where life and limb depends on them, we are responsible for doing the job properly, that's why we get paid.

lbriner 5y ago

That's not defence. It fails the principle of least-surprise. If everyone's experience is that fsync is flushing then why would somebody think to look up the docs for Mac in case they do it differently?

supermatt 5y ago
They do it according to POSIX spec. Linux is the oddball here.
- marcan_42 5y ago
  The POSIX spec is deliberately ambiguous about this.
  - coldtea 5y ago
    So? Did Linux do it like that before? https://lwn.net/Articles/270891/ https://lwn.net/Articles/270891/
    - marcan_42 5y ago
      It does it like that now, which is what I'd expect if I'm writing software.
      - wil421 5y ago
        I’d argue more people develop on non-Linux devices such as Windows and MacOS on linux itself.
        marcan_42 5y ago
        F_FULLFSYNC is nonstandard. As far as I know there is no standard-complicant way to get data on to stable storage on macOS. That's a bit of a problem. It makes a lot more sense to make the standard-compliant way actually sane.
        supermatt 5y ago
        I have said a few times already - F_BARRIERFSYNC. This is likely equivalent to what linux is doing. edit: sorry - not 'standards compliant' (whatever that is - does linux declare support for SIO?), but probably what you are looking for.
        marcan_42 5y ago
        It isn't. I already replied to you above. A barrier does not guarantee data durability and we already know Linux fsync() == macOS F_FULLFSYNC because they have the same (lack of) performance on the same hardware.
        supermatt 5y ago
        Thanks marcan. Apologies for wasting your time.
        deleted 5y ago
        [deleted]
        ryao 5y ago
        I just looked into this, since what you say and what Apple’s documentation says are two different things. Here is Apple’s documentation: https://devstreaming-cdn.apple.com/videos/wwdc/2019/419ef9ipua3hnd2y/419/419_optimizing_storage_in_your_app.pdf?dl=1 https://devstreaming-cdn.apple.com/videos/wwdc/2019/419ef9ip... F_BARRIERFSYNC: fsync() with a barrier F_FULLFSYNC: Drive flush its cache to disk This sounds like the Linux fsync() and Linux syncfs() respectively. What you say is that F_FULLFSYNC is the same as Linux fsync() and your performance numbers back that up. Unfortunately, you would only see a difference between Linux fsync() and Linux syncfs() if you have files being asynchronously written at the same time as the files that are subject to fsync()/syncfs(). fsync() would only touch the chosen files while syncfs() would touch both. If you did not have heavy background file writes and F_FULLSYNC really is equivalent to syncfs(), you would not be able to tell the difference in your tests. That said, let’s look at how this actually works on Mac OS. Unfortunately, the apfs driver does not appear to be open source, but the HFS+ driver is. Here are the relevant pieces of code in HFS+: https://github.com/apple-oss-distributions/hfs/blob/hfs-556.100.11/core/hfs_vnops.c#L3230 https://github.com/apple-oss-distributions/hfs/blob/hfs-556.... https://github.com/apple-oss-distributions/hfs/blob/5e3008b69b8e4e893fb240d3a2f09dbdf7d3c0fb/core/hfs_vfsutils.c#L3419 https://github.com/apple-oss-distributions/hfs/blob/5e3008b6... First, let me start with saying this merits a faceplam. The fsync() operation is operating at the level of the mount point, not the individual file. F_FULLSYNC and F_BARRIERFSYNC are different, but they both might as well be variants of the Linux syncfs(). For good measure, let us look at how this is done on the MacOS ZFS driver: https://github.com/openzfsonosx/zfs/blob/master/module/zfs/zfs_vnops_osx.c#L451 https://github.com/openzfsonosx/zfs/blob/master/module/zfs/z... The file is properly synced independently of the mountpoint, such that other files being modified in the file are not immediately required to be written out to disk. That said, both F_FULLSYNC and F_BARRIERFSYNC on MacOS are mapped by the ZFS driver to the same function that implements fsync() on Linux: https://github.com/openzfs/zfs/blob/master/module/os/linux/zfs/zpl_file.c#L127 https://github.com/openzfs/zfs/blob/master/module/os/linux/z... For good measure, let us look at how syncfs() is implemented by ZFS on Linux: https://github.com/openzfs/zfs/blob/master/module/os/linux/zfs/zpl_super.c#L118 https://github.com/openzfs/zfs/blob/master/module/os/linux/z... It operates on the superblock, which is what MacOS’ HFS+ driver does. From this, I can conclude: Linux syncfs() == macOS F_FULLFSYNC on HFS+ Linux fsync() == macOS fsync()/F_FULLFSYNC/F_BARRIERFSYNC on ZFS Also, MacOS F_BARRIERSYNC is a weakened Linux syncfs() and Apple’s documentation is very misleading (although maybe not technically wrong). POSIX does allow fsync to be implemented via syncfs (sync in POSIX, but I am saying syncfs from Linux to be less confusing). However, not issuing and waiting for the completion of an IO barrier on fsync is broken behavior like you claim. I am not sure how MacOS APFS behaves. I imagine that additional testing that takes into account the nuances in semantics would be able to clarify that. If it behaves like HFS+, it is broken. Edit: Upon further examination and comparing notes with the MacOS ZFS driver team lead, it seems that HFS+ is syncing more than requested when F_FULLFSYNC is used, but less than the entire filesystem. You are fine treating it as a Linux fsync. It is close enough.
olliej 5y ago
Linux only stopped doing the clearly wrong thing in 2008 or so iirc. It is still dumb that there's a definition of fsync() that does not sync :-/
threeseed 5y ago
> why would somebody think to look up the docs It seems reckless to me to not do this when you're interacting with the filesystem using low-level APIs (i.e not via Swift/Obj-C).
masklinn 5y ago
> That's not defence. It fails the principle of least-surprise. Welcome to C APIs in general, and POSIX in particular.
coldtea 5y ago
>That's not defence. It fails the principle of least-surprise. Only if the standard where anything else is a "surprise" is 2022 Linux. Many (all?) other unices and macOS itself since forever work like that. Including Linux itself in the past [1] [1] https://lwn.net/Articles/270891/ https://lwn.net/Articles/270891/
- marcan_42 5y ago
  Drive caches also used to not exist in the past. At that point, behavior was the same as it is on Linux today. It then regressed when drive caches became a thing. Maybe it not being added to OSes when drive caches came into the picture was arguably a bug, and Linux has been the first OS to fix it properly. macOS instead introduced new, non-buggy behavior, and left the buggy one behind :-)
  - acdha 5y ago
    > Drive caches also used to not exist in the past. At that point, behavior was the same as it is on Linux today. It then regressed when drive caches became a thing. You mean in the 1980s? Linux wasn’t used before this wasn’t a concern for sysadmins and DBAs. This concern has been raised for years - back in the PowerPC era the numbers were lower but you had the same arguments about whether Apple had made the right trade-offs, or Linux or Solaris, etc. Given the extreme rarity of filesystem corruption being a problem these days, one might conclude that the engineers who made the assumption that batteries covered laptop users and anyone who cares about this will be using clustering / UPS were correct.
  - CyberRabbi 5y ago
    The minute storage manufacturers introduced drive caches is the minute this bug became the responsibility of storage manufacturers. IMO it’s not the kernel’s responsibility.
    - fomine3 5y ago
      Now Apple is the primary storage manufacturer for Mac.
ecf 5y ago
I’d argue maybe .5% of people are working on something where this is even close to being a concern. Those people probably know what they need to use. Apple doesn’t need to defend anything.
- ClumsyPilot 5y ago
  I am sick of this callous and capricious disrespect for users and their data, rampant throughtout this wanky industry. Do lawyers use Apple computers? Do they work on important documents relating to life and death? Some people have literally been executed because developers couldn't do their job properly. People have been sent to jail for decades because developers fucked up in the british postmaster scandal. Average people life in a dangerous world- work with documents about their financial wellbeing. They live in opressive countries where being gay is punishable by death. They drive 2 ton death machines. And now that we have put computers in places where life and limb depends on them, we are responsible for doing the job properly, that's why we get paid.