5 ms·
In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters. Linux
by tialaramex 5y ago
In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters.
Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you could think of as UTF-16 code units but they don't promise to encode UTF-16.
Probably the files have human readable names, but, maybe not. If you're accepting command line file names it's not crazy to insist on human readable (thus, Unicode) names, but if you process arbitrary input files you didn't create, particularly files you just found by looking around on disks unsupervised - you need to accept that utter gibberish is inevitable sooner or later and you must cope with that successfully.
Rust's OSStr variants match this reality.
- atoav 5y agoThis is what I found quite refreshing about Rust — instead of choosing one of the following: A) The programmer is a almighty god who knows everything, we just expose him to the raw thing B) The programmer is a immature toddler who cannot be trusted, so we handle things for them What Rust does is more among the lines of "you might already know this, but anyways here is a reminder that you, the programmer need to take some decision about this".
- deleted 5y ago[deleted]
- GlitchMr 5y agoFilenames in HFS+ filesystem (an old filesystem used by Mac OS X) are normalized with a proprietary variant of NFD - this is a filesystem feature. APFS removed this feature.
- 1over137 5y ago>APFS removed this feature. And then brought it back. It normalizes now.
- lilyball 5y agoBy “proprietary variant” you mean “publicly documented variant” which IIRC is just the normalization tables frozen in time from an early version of Unicode (the idea being that updating your OS shouldn’t change the rules about what filenames are valid). As for APFS, it ~~doesn’t~~didn’t normalize but I believe it still requires UTF-8. And the OS will normalize filenames at a higher level. EDIT: they added native normalization. At least for iOS, I didn’t dig enough to check it macOS is doing native normalizing or is just normalization-insensitive.
- chrismorgan 5y agoNormalisation is expressly done with the composition of version 3.1 for compatibility: see <https://www.unicode.org/reports/tr15/#Versioning https://www.unicode.org/reports/tr15/#Versioning>. IF that’s what HFS+ does, then “proprietary variant” is wrong. And if not, I’m curious what it does differently. (On the use of version 3.1, note that in practice version 3.2 is used, correcting one typo: see <https://www.unicode.org/versions/corrigendum3.html https://www.unicode.org/versions/corrigendum3.html>.) I find a few references to it being slightly different, but not one of them actually says what’s different; Wikipedia is the only one with a citation (<https://en.wikipedia.org/wiki/HFS_Plus https://en.wikipedia.org/wiki/HFS_Plus>: “and normalized to a form very nearly the same as Unicode Normalization Form D (NFD)[12]”), and that citation says it’s UAX #15 NFD, no deviations. One library that handles HFS+ differently switches to UCD 3.2.0 for HFS+ <https://github.com/ksze/filename-sanitizer/blob/e990e963dc5bd1d172dbc8f85ebd383f6a984ec7/filename_sanitizer/__init__.py#L173-L176 https://github.com/ksze/filename-sanitizer/blob/e990e963dc5b...>, but my impression from UAX #15 is that this should be superfluous, not actually changing anything. (Why is UCD 3.2.0 still around there? Probably because IDNA 2003 needs it: <https://bugs.python.org/issue42157#msg379674 https://bugs.python.org/issue42157#msg379674>.) Update: https://developer.apple.com/library/archive/technotes/tn/tn1150.html#CanonicalDecomposition https://developer.apple.com/library/archive/technotes/tn/tn1... has actual technical information, but the table in question doesn’t show Unicode version changes like they claim it does, so I dunno. Looks like maybe from macOS 10.3 it’s exactly UAX #15, but 8.1–10.2 was a precursor? I’m fuzzy on where the normalisation actually happens, anyway.
- 5y ago
- matja 5y agoZFS can support normalization also: $ echo test > $'\xc3\xb6' $ cat $'\x6f\xcc\x88' cat: ö: No such file or directory $ zfs create -o normalization=formD pool/dataset $ echo test > $'\xc3\xb6' $ cat $'\x6f\xcc\x88' test
- zekica 5y agomacOS is interesting: some APIs normalize filenames while others don't. And it causes some very interesting bugs. One example is when you submit a file in Safari it doesn't normalize the file name while js file.name does.