7 ms·
As a long-time Korean user of macOS, I hate the fact that APFS is normalization-insensitive yet normalization-preserving. As a result, this means that… if you
by goranmoomin 4y ago
As a long-time Korean user of macOS, I hate the fact that APFS is normalization-insensitive yet normalization-preserving.
As a result, this means that… if you create a Korean-named file yourself in Finder: then the file name is normalized in Unicode NFD (decomposed) form, but if you create a Korean-named file in a zsh shell, the file name is normalized in Unicode NFC (composed) form.
You don’t realize the difference until you try an `ls | xxd`, but the bytes are different, even though they look the same.
It’s an invisible mess that people don’t realize until they do. And a lot of programs gets subtly crazy when fopen(A/B) succeeds but listdir(A) doesn’t list B.
- p_l 4y agoSuch issues is why ZFS has optional normalisation support, and serving files to Macs (or generally cross platform) was always one of the major examples of its use
- sneak 4y agoDesigned by Apple in 8859-1
- sacnoradhq 4y agoI think the solution is to stop trying to over-manage user- and filesystem-input. Be faithful and preserve what was entered rather than trying to "second-guess" the user.
- kalleboo 4y agoWhat is the solution here though? They forced normalization in HFS+ and everyone hated that even more since it changed your file names when you copied over files. And they can't be normalization-sensitive because then you can't type in a file name.
- vlovich123 4y agoI still don’t see the problem with normalization.
- hibbelig 4y agoSay that each letter has two versions, and they look exactly alike. This means there are now 8 file names that look like "foo", and it's not clear which one of them you have to type in to get the file you're looking at.
- vlovich123 4y agoThat’s exactly the problem normalization solves - all variants map the same.
- hibbelig 4y agoIm sorry, I misread what you were saying. I understood the opposite of what you were saying.
- CorrectHorseBat 4y agoI guess the issue is that if you copy files between 2 systems, one which does normalization and one which doesn't. Best would be if everyone did normalization, but that ship has sailed :(
- HPsquared 4y agoMaybe each file should have two names instead of one?
- actionfromafar 4y agoMacOS at least has file ID, so there's that.
- kalleboo 4y agoAnd even if both systems have normalization, the normalization changes with Unicode versions, so just different OS versions will cause filename conflicts.
- flohofwoe 4y agoThe solution is simply what Linux does: treat filenames as bags of bytes instead of strings (but this would also include case-sensitivity). At least then the behaviour is completely straightforward and transparent down in the file system layer. UNICODE details like normalization can be handled up in the application layer.
- ClumsyPilot 4y agoDo you speak other languages? Czech language has Á, Č, Ď, Ě, É, Í, Ň, Ó, Ř, Š, Ť, Ů, Ú, Ý Á can be represented in Unicode by its spesific codepoint, or can be made up of two - Letter A and the squiggly bit. So when a user tries to fOpen a file called ÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁ Should by application make 2^32 requests to the file systen to open a file for every possible combination of bytes this steing could be represented as?
- FeepingCreature 4y agoTo be fair, what a sufficiently clever application could do would be to try and naively open the file, note a mismatch, then do a directory walk comparing normalized filenames. Still not great.
- flohofwoe 4y agoIdeally, un-normalized paths wouldn't even make it down to the filesystem layer. I'm just saying that it shouldn't be the filesystem's job to care about text encoding details, but the layers above it (e.g. the filesystem would work in a simple "garbage in - garbage out" mode).
- deleted 4y ago[deleted]
- flohofwoe 4y agoI speak German which has its share of pointy letters :) I'm aware of the normalization problem, but that should be handled in the API layers above the filesystem. In which layer exactly is up for discussion though (but IMHO there should always be a fallback to access files via their "raw bytes path identifiers", if for nothing else than debugging).
- shp0ngle 4y agoBut that’s exactly what they do now?
- Kwpolska 4y agoThis isn't just an issue with the filesystem, apps also randomly produce NFD. And then you copy text from a PDF in Preview to Word, and spellchecking goes haywire.
- wkat4242 4y agoDoes that mean that you could actually have both versions as separate files? That would be really messy. Not to mention possibly exploitable.
- howinteresting 4y agoWell, the ideal solution is for everyone to use NFC all the time. But failing that, having both versions is preferable to what Apple does. What is the threat model that makes having both versions exploitable? Historically, many vulnerabilities (e.g. several git vulns) have come from filesystems doing normalization rather than the lack of it.
- wkat4242 4y agoThe possible exploit is that you could have two versions of a file, one that the user sees and one that is executed. Buy generally anything that leads to unexpected behaviour can be exploited in some way, if not technical then to mislead the user. The question is, why is 'failing that' happening here? And if there's 2 files encoded differently with the same name, how would the user differentiate them?
- deleted 4y ago[deleted]
- howinteresting 4y agoHahahaha, you're at the surface looking at a well of problems that goes all the way to the center of the earth.
- shp0ngle 4y agoI had an issue where I had unicode files on FAT and I wanted to copy them on APFS on mac; and half of them I could not copy, because they were in… uhh either NFD or NFC. One of them. macos just showed some random error when I tried to copy. What I needed to eventually do is to go on Windows, install WSL there, then run some script that recursively renamed them from the first to the other (on the FAT volume), and only then I could copy them from FAT to APFS, on the Mac. (I could probably skip the WSL and do it with some Windows-only tooling, but that would take me even longer.) so this little exercise involved mac, windows, linux (and plan 9, if you count the wsl filesystem shenanigans)
- TheUndead96 4y agoI had no idea this was a thing, this must be infuriating.
- HPsquared 4y agoIs this a bit like how Windows preserves capitalization but doesn't otherwise care about it? (except in WSL of course) - or is it something more complex
- Aaron2222 4y agoAPFS does the exact same thing with capitalisation as well (presuming you're using case-insensitive APFS, which is the default).
- sumtechguy 4y agoWindows has a flag for that for it to respect case. But it would probably bug out a bunch of programs. I usually see it with samba shares. But you can get it to do it on the local filesystem too if you try hard enough. Also it is a bit of a pain when it does happen if you get two files with the 'same name'. One file will be basically invisible to most of the normal windows tools.
- IshKebab 4y agoSimilar except that you can see the difference between lowercase and uppercase.
- messe 4y agoKind of. Imagine if capital letters looked identical to regular letters just with a different byte representation, but Windows preserved them regardless. Well, it turns out that Unicode can represent a lot of characters by way of different possible encodings (for example é could be a single codepoint, or the codepoint of e followed by the codepoint of an acute accent).
- stonekyx 4y agoI think I observed this problem recently too, but with Japanese file names. Basically I had a Git repository and didn’t know about `git config core.precomposeUnicode`. And when the repository is synchronized to a Linux system, on the Linux side it can sometimes have 2 files with the same-looking file name but different normalizations. (Because I think ext4 doesn’t normalize Unicode?) That took me about an afternoon to fix.
- totetsu 4y agoWhat can I do with Japanese characters in Zip files that come out all messed up and Cyrillic when extracted under linux?
- actionfromafar 4y agoIt depends, Japanese filenames are sometimes in an 8-bit codepage. There are several. https://en.wikipedia.org/wiki/Code_page https://en.wikipedia.org/wiki/Code_page https://stackoverflow.com/a/45583116 https://stackoverflow.com/a/45583116 https://github.com/m13253/unzip-iconv https://github.com/m13253/unzip-iconv
- totetsu 4y agoThanks. unzip -O Shift_JIS <file> did the trick.
- monocasa 4y ago> Because I think ext4 doesn’t normalize Unicode? Yeah, native Linux filesystems don't really know anything about unicode at all at their heart. They really work more on raw bytes, simply reserving 0x00 (NUL) and 0x2F ('/'). Anything else goes in a filename as far as the kernel is concerned (including evil stuff like incomplete multibyte sequences, or other invalid UTF-8 byte sequences). User space is welcome to and encouraged to treat filenames as UTF-8 on modern systems, but that's not really enforced anywhere strongly.