24 ms·
APFS does not normalize Unicode filenames
- jacobolus 10y agoThis is a big change. I guess they now decided that compatibility with external systems is a more important goal than end-user-friendliness. It’s a reasonable decision to come to (especially for iOS where end-users don’t ever really interact with the filesystem directly), but it will cause quite a bit of churn in the short term.
- deleted 10y ago[deleted]
- userbinator 10y agoI agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "\003foo\003bar\003baz\012myfile.txt\000"), truly allowing any byte in a filesystem node's name, although that might be going a little bit too far. Things are much easier for the file system if it can just treat names as bags of bytes. If you're really talking about bags (unordered sets), that would certainly make for an interesting filesystem since filename.txt, filemane.txt, and maletent.fix would all be the same...
- kijeda 10y agoIncidentally, your theoretical system is exactly how the DNS stores domain names.
- saghm 10y ago> If you're really talking about bags (unordered sets), that would certainly make for an interesting filesystem since filename.txt, filemane.txt, and maletent.fix would all be the same... I was thinking the same thing! Although if we're being pedantic, I think "bag" is an "unordered multiset". If it were just a plain unordered set, then "filename.txt" would also be the same as "filenam.txt".
- jfim 10y agoIt's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9). If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.
- jstimpfle 10y agoThe problem exists already on the visual level, since there are pairs of distinct Unicode characters that look very much alike. I think the only thing you can really do about that is restricting to ASCII minus control characters. Probably many programmers would be willing to accept that, but non-technical users not. The next best thing would be to enforce a canonical (code-point?) encoding. But given the complexities of Unicode that will get us only so far...
- ttflee 10y agohttps://turnoff.us/geek/user-space-election/ https://turnoff.us/geek/user-space-election/
- jstimpfle 10y agoThat's not about filenames.
- TazeTSchnitzel 10y ago> The problem exists already on the visual level, since there are pairs of distinct Unicode characters that look very much alike. Yes, but software doesn't consider them equivalent.
- int_19h 10y agoThe software also doesn't consider the two strings given as an example above equivalent, and for the same exact reason: they're sequences of different code points.
- ridiculous_fish 10y agoThis feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no... The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's blessed paths /System/Library... and non-blessed paths like ~/Documents/Pokemon.txt. Addressing this properly means reifying that distinction: making apps always be explicit about whether they're working with the user's or the filesystem's view of a file. While we're here, why the should the user even have to name every file? Do you name every piece of junk mail on your desk/kitchen island? The ideal for the user is something like tagging, where naming stuff is optional, names are forgiving and perhaps not unique, and files are not restricted to being in one directory. Meanwhile, filesystem names are an implementation detail and the filesystem enjoys Unicode ignorance. Spotlight moved a bit in this direction, but the end goal is still awfully far away.
- Ericson2314 10y agoI'd argue: Use a langauge with generics and make the path type a ((de-)serializable) black box. You still need to make a decision in the end, but are much more free to change it.
- jstimpfle 10y agoThis is not a language issue.
- Ericson2314 10y agoI didn't say it was. I said generics would help one easily switch their choice on the different semantics, not avoid choosing at all.
- jstimpfle 10y agoMany, or most, languages already do that by representing filenames as native strings (which are usually abstract "Unicode" strings).
- silverwind 10y agoThe same goes for case-sensitivity which, unfortunately, they see as a defect. It would be great for portability if APFS would stay case-insensitive like the Unix file systems, e.g. just a dumb layer that reads/writes bytes.
- inimino 10y agoI think you mean case-aware, not case-sensitive. Just bytes like Unix leads to case-sensitive behavior.
- rurban 10y agoNo. Ever heard about http://websec.github.io/unicode-security-guide/ http://websec.github.io/unicode-security-guide/ Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out. There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed scripts or confusables. I summarized a few problems at http://perl11.org/blog/unicode-identifiers.html http://perl11.org/blog/unicode-identifiers.html
- jwilk 10y ago> No. Ever heard about http://websec.github.io/unicode-security-guide/ http://websec.github.io/unicode-security-guide/ Which part of it is relevant to pathnames? > To the canonical form of course Do you mean NFC? > not the crazy Python 3 or Apple idea of NFD. In what circumstances does Python 3 normalize pathnames? > If it's encoded as bytes you get garbage in - garbage out. What do you mean by "encoded as bytes"? > http://perl11.org/blog/unicode-identifiers.html http://perl11.org/blog/unicode-identifiers.html This is about programming language identifies, not about pathnames, which are very different beasts.
- rurban 10y ago> Which part of it is relevant to pathnames? All parts regarding identifiers and names, if you see pathnames as publicly identifiable information. See below. > In what circumstances does Python 3 normalize pathnames? None. Python 3 normalizes identifiers to the shorter non-canonical NFKC form. Apple HFS+ normalizes pathnames to the longer D canonical form. Which is faster but takes more space. Usually space is more important than CPU. > What do you mean by "encoded as bytes"? What everyone else means on byte encodings: 1-1 mapping of bytes, without any further knowledge of the encoding. > This is about programming language identifies, not about pathnames, which are very different beasts. Partially. pathnames can also be argued to be identifiers, similar to domain names, email names, user names, language identifiers. Apple does so. Ask plan9 what they think about pathname semantics. Not everyone is in the garbage in - garbage out camp. Many systems do encode pathnames and have character restrictions. Esp. important is the popular / spoof, which people from the garbage camp don't care about. Julia is recent proponent of the garbage camp, btw.
- sametmax 10y agoHow do you normalize arab or chinese in a meaningful way for people speaking these languages ?
- Asooka 10y agoYou don't, there is only one unicode representation of those characters.
- sametmax 10y agoSo what does mac do with those ?
- icebraining 10y agoUse that representation? The problem only happens when there is more than one way to represent the same text.
- sametmax 10y agoOk
- huxley 10y agoInterestingly that created its own annoyances for some since normalization occurs at a different stage. Unicode and UTF-8 unify many Traditional & Simplified Chinese Characters as well as Japanese Kanji and Korean Hanja. There were some extremely angry reactions to Han Character Unification[1], though efforts have been made to overcome many of the initial complaints. [1] "Why Unicode Won’t Work on the Internet: Linguistic, Political, and Technical Limitations" http://www.hastingsresearch.com/net/04-unicode-limitations.shtml http://www.hastingsresearch.com/net/04-unicode-limitations.s...
- jrochkind1 10y agoI don't believe that's literally true, some all legal and correct Japanese unicode sequences will be transformed under at least some of the unicode normalization forms, no?
- quotemstr 10y agoNo good has ever come from allowing BEL and DEL as part of filenames. I'm with David Wheeler: https://www.dwheeler.com/essays/fixing-unix-linux-filenames.html https://www.dwheeler.com/essays/fixing-unix-linux-filenames.... We need to limit filenames for the good of the entire system and the whole community. Filenames as byte strings may sound good, but nobody ever thinks of the costs and the scant benefits.
- YSFEJ4SWJUVU6 10y agoI disagree completely. What we need is to allow even slashes in the file names – not restrict (for what an end user may feel arbitrarily) even more writeable characters off file names, just so we can avoid fixing our replacing our tools or writing one line more in a shell script.
- coldtea 10y agoAny solution you come up with must work with replacing existing software piecemeal, or it's no solution.
- silvestrov 10y ago> Things are much easier for the file system if it can just treat names as bags of bytes. And much, much harder for applications if that "bag of bytes" is an invalid UTF-8 sequence. You will end up with an invalid string (or an exception), and trying to open that file will then fail. I'd really hope that Apple checks that the filenames are valid UTF-8 as they otherwise can end up with very interesting security bugs.
- asdfaoeu 10y agoYou can handle that fine provided you don't assume that. > I'd really hope that Apple checks that the filenames are valid UTF-8 as they otherwise can end up with very interesting security bugs. I think that's a bit pessimistic as Linux has run for year like that without any real issues.
- TazeTSchnitzel 10y ago> And much, much harder for applications if that "bag of bytes" is an invalid UTF-8 sequence. It makes copying-and-pasting filenames as Unicode unreliable, for one thing.
- loeg 10y agoAt least valid utf-8 isn't difficult for a filesystem to enforce.
- hawski 10y agoThis sounds like DJB's Netstring [1]. "hello world!" encoded as "12:hello world!," An empty string as "0:," [1] https://en.wikipedia.org/wiki/Netstring https://en.wikipedia.org/wiki/Netstring
- tyingq 10y ago>perhaps with one or two exceptions like '/' and \0 Aren't colons (:) an issue with MacOS as well? I think Finder and other userspace apps convert them to slashes. I suppose though, there's a case for the filesystem not caring about that.
- infogulch 10y agoIf I were allowed one more distinction I would definitely add utf-8 only.
- emn13 10y agoI'm not so sure: the thing is, those strings are not just a bag of bytes. They have semantics. Is it legal to include bytes that are 0? How about slashes in file names? How about byte sequences that aren't utf8 at all? These things aren't just byte streams, they have semantics. So one risk of treating this as "just bytes" is that bugs will introduce byte sequences that aren't utf8 at all, which will cause other programs to fail, or worse, to "try" and thereby corrupt the data further. Another risk is that since it's supposed to be utf-8, some programs may do canonicalization internally to avoid confusing situations. This may even happen accidentally (though it's not likely): after all, a unicode-processing system could be forgiven for transparently changing canonicalization. But if a program canonicalizes you can now get really weird behavior such as opening a file, then saving it, and ending up with two files that look identical - because the write wrote to a path that was canonicalized. Additionally, even though you can never avoid confusing paths entirely without considering the glyphs rendered, you are losing a very simple check against a fairly large class of errors.
- deleted 10y ago[deleted]
- QuercusMax 10y agoThis seems especially bad because US-based developers who don't test with unicode filenames might not come across this issue, leaving all their non-English-speaking customers broken. (Not that this excuses such developers in any way.) It also means that, yet again, every app will need to be updated for a new version of iOS. Makes me wonder how many apps will be left behind if not updated? Thousands? Hundreds of thousands? Millions?
- spullara 10y agoExcept filesystem experts like Dropbox, developers probably shouldn't be letting users name their files.
- andybak 10y agoI'm not sure I understand. How can an app NOT allow me to name my files - unless you're talking about some iOS-like "hide the filesystem" silo.
- deleted 10y ago[deleted]
- xenadu02 10y agoYou should not be using anything other than UUIDs or integers for file names. Maintain your own mapping in a database or file. Using a network value or a value returned by an API is just asking for trouble. If a user names a file that will be hidden behind a URL the same advice applies. If not then the user can use any sequence of bytes they want and you shouldn't care.
- astrange 10y agoIt's poor UI to not name a file on the user's own machine, if they'll ever have to look at it. I'm counting things like web browser caches in this because OmniDiskSweeper/etc users will be seeing the large files you put in there.
- 10y ago
- deleted 10y ago[deleted]
- lathiat 10y agoSeems to me that Apple would be smart to hook all of the file functions and survey and/or alert on this situation somehow. I only just learnt about this unicode normalisation recently looking at ZFS which has options for it I had never seen until reading the Ubuntu Root FS on ZFS guides which talk about setting it.
- djrogers 10y agoiOS 10.3 with APFS has been in public and developer beta for several months - it's up to beta 7 right now in fact. If this were as vast a problem as Micheal Tsai presents in this post, wouldn't we (the devs and beta testers) be running in to this a lot? Given how loudly the tech press proclaims any perceived mis-step by Apple, I'd have to believe we'd have been reading tons of 'Apple is Doooooooomed' articles about this by now. Given that this hasn't happened, and I haven't seen similar problem reports on dev forums and other hangouts, I'd lean towards there being some miscommunication or misunderstanding here.
- williamscales 10y agoIs it possible that the beta testers are largely in the US and so wouldn't have seen this issue much?
- eridius 10y agoThis is not likely to be a particularly big issue on iOS, because the file system isn't directly exposed to the user (and therefore the user can't go making changes behind the app's back). There are of course still edge cases that could cause a problem, but they're going to be relatively rare. But this may become a much bigger issue when we start using APFS on macOS.
- mattcurtis 10y agoThis isn't true of apps such as viewers/readers that use App File Sharing.
- bartvk 10y agoAs I understand it, it may only be a problem when using the straight libc calls. But most devs just use the file system stuff that's in Apple's Foundation framework.
- therealmarv 10y agoIs APFS still using Apple's style UTF-8 for e.g. Umlauts? I had a lot of trouble with rsync and also Samba later (filenames and folders hidden) when I discovered that Umlauts on HFS are different than Umlauts on e.g. Ext4.
- yuhong 10y agoThat is due to the HFS+ normalization that APFS eliminates. The filename is stored in UTF-16 and OS X converts it to UTF-8.
- frik 10y agoIs it stored "real UTF-16" or in USC2 wide-chars like in NTFS? The later would be a big problem.
- yuhong 10y agoI believe HFS+ was created before Mac OS supported surrogates.
- eridius 10y agoNormalization and UTF-16 are completely orthogonal. It doesn't matter in the slightest whether the filesystem stores the filename in UTF-16 or UTF-8, either way it's still a sequence of Unicode scalar values, and that's what matters.
- kalleboo 10y agoHFS does a lot more than convert to UTF-8. E.g. it will normalize ö to o+(umlaut), using custom normalization rules that nobody else uses. If you tell HFS to store a file with an Ö in it, and then list the directory, the character Ö is nowhere to be found.
- eridius 10y agoWhat do you mean by "different"? Umlauts on HFS+ are still Umlauts anywhere else. The only real oddity of HFS+ (beyond the fact that it does normalization at all) is that it's not using NFD, it's using a variant of NFD based on an old version of Unicode (it has to be this way because the normalization tables must be immutable or there's compatibility issues when reading drives written to from different versions of the OS). So if you take a filename and convert it to NFD, it may not be the exact same byte sequence that you get if you plug that filename into HFS+ (but in most cases it will be). But whatever byte sequence you get from HFS+ is still going to be a valid Unicode sequence.
- alphabettsy 10y agoWouldn't this be seen as an issue in betas? I haven't seen anything indicating this is widespread so far? Why would that be, just not wide enough deployment yet?
- Moru 10y agoMabe most beta testers are based in english-speaking countries and countries where most people are used to stay away from non-english characters and never noticed the problem? I live in Sweden and still avoid using åäö in filenames because of old habits from DOS/Atari era.
- kzrdude 10y agoIt's sad when we give up to technological limitations! They are our tools, not the other way around ;-)
- Prego 10y agoI've been very excited about ReFS -- a real "modern" Filesystem that leaves legacy issues behind. We've been using it for large storage systems, and am hoping it will become a viable solution for everything soon. It solves most of these issues.
- kupiakos 10y agoExcept cross-platform support, right?
- asdfgadsfgasfdg 10y agoIf Microsoft open source it with a free patent license it might be useful. Until that it is just another proprietary FS locked to a single OS.
- Prego 10y agoLike APFS?
- asdfgadsfgasfdg 10y agoYep.
- quotemstr 10y agoA "modern" filesystem that lacks support for hard links? No thanks.
- makecheck 10y agoTechnically the presentation-layer problem existed already with things like legacy path separators, making the Finder tell lies in the presence of colons or slashes. I suspect that normalization differences will be a little like telling two files apart when one has a trailing space, or hidden file extensions; there will have to be some distinction but maybe no easy answer.
- kalleboo 10y agoLinus Thorvalds will be happy to hear that http://www.cio.com/article/2868393/linus-torvalds-apples-hfs-is-probably-the-worst-file-system-ever.html http://www.cio.com/article/2868393/linus-torvalds-apples-hfs...
- ben_bai 10y agoHFS+ can be configured at creation time to be case sensitive. I did so a while back. Worked perfectly except for one application which could not find it's files. So i had to create a container and Format it case in sensitive and intall the APP there...
- cjensen 10y agoAdobe apps are notoriously incompatible with case sensitivity. If your apps work, that's great. But if Apple switched to case sensitivity by default, it would break apps.
- slrz 10y agoApple breaks apps all the time on macOS.
- __david__ 10y agoI used to format my whole disk as case-insensitive but enough poorly written apps broke (ok, mostly Steam) that I now just have a case sensitive partition for my source code and /usr/local.
- deleted 10y ago[deleted]
- cjensen 10y agoThis is for iOS, where the app developer fully controls file naming within their sandbox. It is very unlikely that MacOS will fail to normalize because filenames there are presented directly to the user.
- nailer 10y agoFor anyone else wondering: https://en.m.wikipedia.org/wiki/Unicode_equivalence https://en.m.wikipedia.org/wiki/Unicode_equivalence
- eric_h 10y agoThank you, that's actually a very well written Wikipedia article. I'd not heard the term "normalization" used wrt Unicode before.
- deleted 10y ago[deleted]
- Sami_Lehtinen 10y agoUnicode isn't required to mess up things. Here's what baffled me for a while with NTFS. I'm pretty sure these issues are well known. http://www.sami-lehtinen.net/blog/linux-windows-ntfs-differences-and-potential-problems http://www.sami-lehtinen.net/blog/linux-windows-ntfs-differe...
- std_throwaway 10y agoPeople should not use a non-compliant file system driver to create corrupted entries. NTFS mounts are for windows machines only.
- vetinari 10y agoWhat exactly makes a FS driver compliant or non-compliant? Is there a NTFS compliance test suite, that we can run against specific implementations? If there isn't, accusations of non-compliance are just FUD.
- dom0 10y agoThis is not a bug in either, it's a configuration defect. NTFS-3g used with the correct options, namely windows_names, works exactly as expected, but is then not POSIX compliant anymore.
- vbezhenar 10y agoI'm not sure if normalization is good idea (generally because Unicode is complex beast and moving that complexity inside a kernel should be carefully weighted), but I'm sure that it doesn't solve any real problem. Characters "A" and "А" looks identical, unless you're missing Cyrillic font, but they won't be normalized, because they are completely different characters. There are many more other visually identical strings. So while normalization might solve some simple problems, it's not a complete solution, so filesystem might just treat names as byte arrays and let user solve his problems.
- TazeTSchnitzel 10y agoConfusable characters look similar or the same to humans. Canonically equivalent Unicode sequences look the same to machines. The latter is a much more significant problem, because it can wreak havoc with interoperability.
- acdha 10y agoVery true - and it's amplified by inconsistency allowing problems to spread further before being noticed. At work I deal with a lot of Bag-It archives where we have a text manifest of checksums accompanying files on disk and this reliably bites users of tools when something (archive or network transfer tool, Git or SVN, transition to/from a Mac with HFS+, etc.) causes the encoding in the manifest not to match the local filesystem, and the confusion is amplified because some tools will handle normalization differences so the bug report is “why does tool A say this file is missing when Explorer/Finder and tool B say it's fine?”
- slrz 10y agoCanonically equivalent Unicode sequences look the same to machines. Memcmp disagrees, as do the default equality operators of most programming languages in existence.
- TazeTSchnitzel 10y agoSure, but normalisation can nonetheless happen automatically and implicitly in many places.
- al2o3cr 10y agoMore generally, once APFS is deployed users can legitimately end up with multiple files in the same folder whose names only differ in normalization. The initial message that starts this off seems to imply the opposite - instead, application developers should be normalizing the name before handing it to the filesystem. In that case, an application which allowed non-normalized naming would arguably have a bug.
- Someone 10y agoAsking applications to enforce an OS-wide policy is asking for problems, even ignoring the existence of malware. You wouldn't say "the OS doesn't check file attributes to see whether you are allowed to access a file; that's left to applications", either.
- rick_cheese 10y agoThe way iOS abstracts the filesystem away from user-view makes this less of an issue than it otherwise would be but still a good find by the author, as an aside surely I'm not the only one who thought of [1] when I read "APFS now treats all files as a bag of bytes on iOS" ;) [1] https://www.youtube.com/watch?v=OT7xc_XqYO8 https://www.youtube.com/watch?v=OT7xc_XqYO8
- kalleboo 10y agoDon't Mac apps already have to deal with network and FAT32 drives? Or does macOS already normalize those?
- m-j-fox 10y agoI want to make the gzipped contents of my file it's name and leave the actual contents blank or whatever metadata. Thanks apfs!
- cesarb 10y agoThis used to be a pain point with git, when some developers were using MacOS and the repository had file names with accents; to git, it looked like the files had been renamed. Some time later, git added the "core.precomposeunicode" option to work around this problem.
- killercup 10y agoNot sure if APFS has such a thing, but I think I heard about it a while back: Could they introduce a directory-level option to automatically normalize all files below that node? (Same with case-sensitivity, which I think Adobe software still has problems with.)
- dom0 10y agoIt doesn't really matter what they do, since filesystem naming is FUBAR and has been FUBAR pretty much since UNIXv1, and possibly even earlier than that outside the UNIX family.