4 ms·

The title is click-bait and over-dramatises the issue. The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to c

The title is click-bait and over-dramatises the issue. The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here). If you want to see the kind of issues that path normalisation brings, check out this: https://github.com/thibaudgg/rb-fsevent/blob/master/ext/fsevent_watch/FSEventsFix.c https://github.com/thibaudgg/rb-fsevent/blob/master/ext/fsev... I'd like to believe that most developer would prefer the current behaviour over that proposed by the article (normalisation). If there are issues in Apple's Finder or other high level APIs, I'd like to believe they will be fixed before the final release. These are fixable bugs, IMHO.

the_mitsuhiko 9y ago
> The title is click-bait and over-dramatises the issue. Sadly it's not. > The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here). That is incorrec. APFS treats filenames as utf-8 strings and depending on the API you are using normalization is still taking place but on different levels. For instance all Cocoa APIs will perform normalization but the underlying syscalls and posix APIs will not.
- bigin 9y ago
  So it is correct then, the file system doesn't concern itself with normalization. File names are stored internally as utf-8 strings which are just a sequence of bytes.
  - the_mitsuhiko 9y ago
    By that logic that was true for HFS+ as well.
    - bigin 9y ago
      HFS+ did perform normalization on a file system level and stored strings as utf-16 so it's not true for HFS+.
      - the_mitsuhiko 9y ago
        Pretty sure normalization was done in open() and not the fs. Normalization was performed even for fat32 sticks.
        bigin 9y ago
        So you disagree with the linked blog post then? I mean this entire discussion is based on the premise that HFS+ performs normalization and APFS does not. The disrcepancy means that what used to happen automatically is now the responsibility of individual applications and the author hypothises that this means that shit will break.
        the_mitsuhiko 9y ago
        Not sure what you want me to say. In my mind the current situation with APFS is a massive pain and was not well thought through. Previously normalization (while terrible) was at least consistent. Now different APIs have widely different behaviour and it's even worse on the application level. What used to be a annoyance has now become absolute chaos with no guidance from apple. My understanding of the article is effectively: AFPS on macOS in the current form is unusable for non-English users and I strongly agree with that. While it might be partially usable, in particular if you upgrade from an earlier mac was performed, it has a ton of really terrible edge cases most users will quickly run into.
  - Someone 9y ago
    Every UTF-8 string is a sequence of bytes, but not every sequence of bytes is a UTF-8 string.
  - cryptonector 9y ago
    A terrible mistake. APFS should have been normalization-preserving/normalization-insensitive.
    - chipotle_coyote 9y ago
      Quoting from https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html https://developer.apple.com/library/content/documentation/Fi... "The case-insensitive variant of APFS is normalization-preserving, but not normalization-sensitive."
      - deleted 9y ago
        [deleted]
      - cryptonector 9y ago
        "The case-sensitive variant of APFS is both normalization-preserving and normalization-sensitive." It should just always be form-preserving and form-INsensitive. There is absolutely no reason to want the combination they give. It makes me suspect that they didn't bother separating case- and form-insensitivity, although I can't imagine why one wouldn't. It seems like.. a mistake. Someone misunderstood.
    - deleted 9y ago
      [deleted]
kalleboo 9y ago
The problem seems to come from the high-level APIs doing normalization. So if you have two files in a directory, one normalized and one not-normalized and open the non-normalized file, the high-level API will then normalize that filename and you open the wrong file. I've always disliked the practice of messing around with file paths (storing them, concatenating them, etc). I preferred the way that the Classic MacOS typically dealt with filesystem references and aliases instead of file paths. This also meant users can rename or move files and references still work.
- deleted 9y ago
  [deleted]
- gtufano 9y ago
  I understand that the bookmark feature of NSURL is (more or less, but most more) the same. cit: "Whereas path- and file reference URLs are potentially fragile between launches of your app, a bookmark can usually be used to re-create a URL to a file even in cases where the file was moved or renamed."
- rndgermandude 9y ago
  Having two sets of APIs, one that does not mess with paths and one that does or adds other constraints, will cause all kinds of funky problems. For APFS it will be high-level-API using applications not being able to open certain files or using the wrong file[0], like stated. You got a similar siltation with NTFS where the file system supports long paths while the userland WinAPI does use a much smaller length limit (unless you jump through "\\?\ hoops, which does not work for relative paths), which renders certain files "unopenable" by certain applications. [0] Regarding opening the wrong files, anybody remember the Android zip vulnerabilities? https://googlesystem.blogspot.com/2013/07/the-8219321-android-bug.html https://googlesystem.blogspot.com/2013/07/the-8219321-androi... It's not that hard to imagine that some macOS software does security/sanity checks on files using the low-level non-normalized API but then opens the (wrong and unchecked) file later with normalizing high level API, or vice versa. Having this API discrepancy built into your OS certainly makes these kinds of things more likely.
Someone 9y ago
"The choice of APFS is that a filename is a sequence of bytes" I don't think that is true. https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html#//apple_ref/doc/uid/TP40016999-CH6-DontLinkElementID_3 https://developer.apple.com/library/content/documentation/Fi...: "Filenames in APFS are encoded in UTF-8 and aren’t normalized." So, names must be UTF-8; not every sequence of bytes is valid UTF-8. What they removed from the file system are the Unicode normalization tables.
cryptonector 9y ago
The problem is serious if you run into it. The FS needs to be normalization-preserving/insensitive. http://cryptonector.com/2010/04/on-unicode-normalization-or-why-normalization-insensitivity-should-be-rule/ http://cryptonector.com/2010/04/on-unicode-normalization-or-...

the_mitsuhiko 9y ago

> The title is click-bait and over-dramatises the issue. Sadly it's not. > The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here). That is incorrec. APFS treats filenames as utf-8 strings and depending on the API you are using normalization is still taking place but on different levels. For instance all Cocoa APIs will perform normalization but the underlying syscalls and posix APIs will not.

bigin 9y ago
So it is correct then, the file system doesn't concern itself with normalization. File names are stored internally as utf-8 strings which are just a sequence of bytes.
- the_mitsuhiko 9y ago
  By that logic that was true for HFS+ as well.
  - bigin 9y ago
    HFS+ did perform normalization on a file system level and stored strings as utf-16 so it's not true for HFS+.
    - the_mitsuhiko 9y ago
      Pretty sure normalization was done in open() and not the fs. Normalization was performed even for fat32 sticks.
      - bigin 9y ago
        So you disagree with the linked blog post then? I mean this entire discussion is based on the premise that HFS+ performs normalization and APFS does not. The disrcepancy means that what used to happen automatically is now the responsibility of individual applications and the author hypothises that this means that shit will break.
        the_mitsuhiko 9y ago
        Not sure what you want me to say. In my mind the current situation with APFS is a massive pain and was not well thought through. Previously normalization (while terrible) was at least consistent. Now different APIs have widely different behaviour and it's even worse on the application level. What used to be a annoyance has now become absolute chaos with no guidance from apple. My understanding of the article is effectively: AFPS on macOS in the current form is unusable for non-English users and I strongly agree with that. While it might be partially usable, in particular if you upgrade from an earlier mac was performed, it has a ton of really terrible edge cases most users will quickly run into.
- Someone 9y ago
  Every UTF-8 string is a sequence of bytes, but not every sequence of bytes is a UTF-8 string.
- cryptonector 9y ago
  A terrible mistake. APFS should have been normalization-preserving/normalization-insensitive.
  - chipotle_coyote 9y ago
    Quoting from https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html https://developer.apple.com/library/content/documentation/Fi... "The case-insensitive variant of APFS is normalization-preserving, but not normalization-sensitive."
    - deleted 9y ago
      [deleted]
    - cryptonector 9y ago
      "The case-sensitive variant of APFS is both normalization-preserving and normalization-sensitive." It should just always be form-preserving and form-INsensitive. There is absolutely no reason to want the combination they give. It makes me suspect that they didn't bother separating case- and form-insensitivity, although I can't imagine why one wouldn't. It seems like.. a mistake. Someone misunderstood.
  - deleted 9y ago
    [deleted]

kalleboo 9y ago

The problem seems to come from the high-level APIs doing normalization. So if you have two files in a directory, one normalized and one not-normalized and open the non-normalized file, the high-level API will then normalize that filename and you open the wrong file. I've always disliked the practice of messing around with file paths (storing them, concatenating them, etc). I preferred the way that the Classic MacOS typically dealt with filesystem references and aliases instead of file paths. This also meant users can rename or move files and references still work.

deleted 9y ago
[deleted]
gtufano 9y ago
I understand that the bookmark feature of NSURL is (more or less, but most more) the same. cit: "Whereas path- and file reference URLs are potentially fragile between launches of your app, a bookmark can usually be used to re-create a URL to a file even in cases where the file was moved or renamed."
rndgermandude 9y ago
Having two sets of APIs, one that does not mess with paths and one that does or adds other constraints, will cause all kinds of funky problems. For APFS it will be high-level-API using applications not being able to open certain files or using the wrong file[0], like stated. You got a similar siltation with NTFS where the file system supports long paths while the userland WinAPI does use a much smaller length limit (unless you jump through "\\?\ hoops, which does not work for relative paths), which renders certain files "unopenable" by certain applications. [0] Regarding opening the wrong files, anybody remember the Android zip vulnerabilities? https://googlesystem.blogspot.com/2013/07/the-8219321-android-bug.html https://googlesystem.blogspot.com/2013/07/the-8219321-androi... It's not that hard to imagine that some macOS software does security/sanity checks on files using the low-level non-normalized API but then opens the (wrong and unchecked) file later with normalizing high level API, or vice versa. Having this API discrepancy built into your OS certainly makes these kinds of things more likely.

Someone 9y ago

"The choice of APFS is that a filename is a sequence of bytes" I don't think that is true. https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html#//apple_ref/doc/uid/TP40016999-CH6-DontLinkElementID_3 https://developer.apple.com/library/content/documentation/Fi...: "Filenames in APFS are encoded in UTF-8 and aren’t normalized." So, names must be UTF-8; not every sequence of bytes is valid UTF-8. What they removed from the file system are the Unicode normalization tables.

cryptonector 9y ago

The problem is serious if you run into it. The FS needs to be normalization-preserving/insensitive. http://cryptonector.com/2010/04/on-unicode-normalization-or-why-normalization-insensitivity-should-be-rule/ http://cryptonector.com/2010/04/on-unicode-normalization-or-...