12 ms·

Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of by

Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.

pavlov 5y ago
The most common desktop file systems are case-insensitive, which complicates the picture.
- Pxtl 5y ago
  Still, it looks like the right thing to do is let the filesystem do the filesystem's job. The filesystem should be normalizing unicode and enforceing the case-insensitivity and whatnot, but just the filesystem. Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes.
  - dataflow 5y ago
    I'm not sure this problem even has a "right" solution. > Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes. What do you do when the input isn't a dumb pile of bytes, but actual text? (Like from a text box the user typed into?)
    - rzzzt 5y ago
      Maintain a table that maps the original file name to random-generated one that doesn't hit these gotchas.
      - rob_c 5y ago
        And place the files in chunks, and... Wait I think we're getting close to reinventing block storage again ;)
      - dataflow 5y ago
        I'm afraid I don't follow. Who maintains this table and who consumes it? What if they're different entities? How do you prevent it from going out of sync with the file system when the user renames a file? Are you inventing your own file system here? How do you deal with existing file systems?
        rzzzt 5y ago
        I assumed that you have a system where file management/synchronization happens strictly through a web interface, and files are not changed or renamed outside this system's knowledge. Under these preconditions, having such a mapping table frees the users from having to abide whatever restrictions the underlying file system places on valid file names.
        dataflow 5y ago
        Oh I was talking about the general case from a programming standpoint. What do you do on a typical local filesystem? The point I'm trying to get at being, you need to worry about the representation at multiple layers, not just at the bottom FS layer.
- mjevans 5y ago
  Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.
  - laurent92 5y ago
    So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? And John.McCauley@yahoo.fr and john.mccauley@yahoo.fr being attributed to two different people ;)
    - justaguy37 5y ago
      can we just say no to capital letters? (or lowercase?) do capital letters have a good enough usage case to justify their continued existence?
      - colejohnson66 5y ago
        You are free to stop using capital letters, but good luck getting everyone to go along. Capitals have been around for centuries (they’re older than the printing press) and aren’t going anywhere.
        bmn__ 5y ago
        The lower-case letters in Greek/Latin/Cyrillic are the new additions, initially we only had what is now called upper-case.
      - Lammy 5y ago
        Fun fact: The Apple Ⅱ and Ⅱ+ originally only did upper-case, and it was very popular to add a Shift Key / lower-case mod via one of the gamepad buttons: https://web.archive.org/web/20010212094858/http://home.swbell.net/rubywand/Csa2KBPADJS.html#001 https://web.archive.org/web/20010212094858/http://home.swbel...
    - vgel 5y ago
      First one: yes, though good UI should prevent it from happening unless the user really intended it (for example I have ~/Documents symlinked into Dropbox, so ~/documents could be local-only documents) Second one: no, emails are not filenames, and more generally distinguishability is more important for identifiers. In cases where identifiers like emails need to be mapped to filenames, like caches, they should be normalized.
arka2147483647 5y ago
That works for programmers, but not for users. There could be several files with the same name, buth with different encodings. Worse, depending on how your terminal encodes user input, some of them migth not be typable.
- zarzavat 5y ago
  From the users perspective I don't want any normalisation at all. It's good as long as you only have one file system but as soon as you get multiple file systems with conflicting rules (which includes transferring files to other people) it becomes hell. Unfortunately we are stuck with that hell.
tialaramex 5y ago
In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters. Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you could think of as UTF-16 code units but they don't promise to encode UTF-16. Probably the files have human readable names, but, maybe not. If you're accepting command line file names it's not crazy to insist on human readable (thus, Unicode) names, but if you process arbitrary input files you didn't create, particularly files you just found by looking around on disks unsupervised - you need to accept that utter gibberish is inevitable sooner or later and you must cope with that successfully. Rust's OSStr variants match this reality.
- atoav 5y ago
  This is what I found quite refreshing about Rust — instead of choosing one of the following: A) The programmer is a almighty god who knows everything, we just expose him to the raw thing B) The programmer is a immature toddler who cannot be trusted, so we handle things for them What Rust does is more among the lines of "you might already know this, but anyways here is a reminder that you, the programmer need to take some decision about this".
  - deleted 5y ago
    [deleted]
- GlitchMr 5y ago
  Filenames in HFS+ filesystem (an old filesystem used by Mac OS X) are normalized with a proprietary variant of NFD - this is a filesystem feature. APFS removed this feature.
  - 1over137 5y ago
    >APFS removed this feature. And then brought it back. It normalizes now.
stefan_ 5y ago
Sure but at some point you might want to create a file and frequently using user input or filter files using some user provided query string, the kind of use cases that unicode normalization was invented for. So the whole "opaque blob of bytes" filesystem handling is nice if all you want is to not silently corrupt files, but it is very obviously not even covering 10% of normal use cases. Rust isn't being super smart, it just has its hands thrown up in the air.
alkonaut 5y ago
Falls over on the fact that I don’t want to be able to write these two files in the same dir. if I write file ö1.txt and ö1.txt then I want to be warned that the file exists even of the encoding is different when I use two different apps but try to write the same file. The same applies for a.txt and A.txt on case insensitive file systems (as someone pointed out the most common desktop file systems are).
jrochkind1 5y ago
Well, precisely because if you don't normalize the filenames, ö ≠ ö. You could have two files with different filenames, `göteborg.txt` and `göteborg.txt`, and they are different files with different filenames. Or you could have one file `göteborg.txt`, and when you try to ask for it as `göteborg.txt`, the system tells you "no file by that name". Unicode normalization is the solution to this. And the unicode normalization algorithms are pretty good. The bug in this case is that the system did not apply unicode normalization consistently. It required a non-default config option to be turned on to do so? I don't really understand what's going on here, but it sounds like a bug in the system to me that this would be a non-default config option. Dealing with the entire universe of human language is inherently complicated. But unicode gives us some actually pretty marvelous tools for doing it consistently and reasonably. But you still have to use them, and use them right, and with all software bugs are possible. But I don't think you get fewer crazy edge cases by not normalizing at all. (In some cases you can even get security concerns, think about usernames and the risk of `jöhn` and `jöhn` being two different users...). I know that this is the choice some traditional/legacy OSs/file systems make, in order to keep pre-unicode-hegemony backwards compat. It has problems as well. I think the right choice for any greenfield possibilities is consistent unicode normalization, so `göteborg.txt` and `göteborg.txt` can't be two different files with two different filenames. [btw I tried to actually use the two common different forms of ö in this text; I don't believe HN normalizes them so they should remain.]
- nieve 5y ago
  It looks like instead of the config option switching everything to use the same normalization it keeps a second copy of the name in a database to compare to. What a horrible kludge, I wonder how they even got into this situation of using different normalization in different parts of the system?
  - jrochkind1 5y ago
    That seems an odd choice indeed, because even if you do have different normalizations in differnet parts of the system, you don't need to keep multiple copies -- you just need to apply the right normalization in the right place. All of the unicode normalization algorithms are both idempotent and of course completely deterministic. If you apply NFD to any legal input, you get the same thing every time -- there's no need to store the NFC version separately to compare it to NFC input when all you have is NFD otherwise, you can just normalize the input to NFD to compare it to what you have! Unless it was meant to be for performance?

pavlov 5y ago

The most common desktop file systems are case-insensitive, which complicates the picture.

Pxtl 5y ago
Still, it looks like the right thing to do is let the filesystem do the filesystem's job. The filesystem should be normalizing unicode and enforceing the case-insensitivity and whatnot, but just the filesystem. Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes.
- dataflow 5y ago
  I'm not sure this problem even has a "right" solution. > Wrappers around it like whatever Nextcloud is doing should be treating the filenames as a dumb pile of bytes. What do you do when the input isn't a dumb pile of bytes, but actual text? (Like from a text box the user typed into?)
  - rzzzt 5y ago
    Maintain a table that maps the original file name to random-generated one that doesn't hit these gotchas.
    - rob_c 5y ago
      And place the files in chunks, and... Wait I think we're getting close to reinventing block storage again ;)
    - dataflow 5y ago
      I'm afraid I don't follow. Who maintains this table and who consumes it? What if they're different entities? How do you prevent it from going out of sync with the file system when the user renames a file? Are you inventing your own file system here? How do you deal with existing file systems?
      - rzzzt 5y ago
        I assumed that you have a system where file management/synchronization happens strictly through a web interface, and files are not changed or renamed outside this system's knowledge. Under these preconditions, having such a mapping table frees the users from having to abide whatever restrictions the underlying file system places on valid file names.
        dataflow 5y ago
        Oh I was talking about the general case from a programming standpoint. What do you do on a typical local filesystem? The point I'm trying to get at being, you need to worry about the representation at multiple layers, not just at the bottom FS layer.
mjevans 5y ago
Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.
- laurent92 5y ago
  So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? And John.McCauley@yahoo.fr and john.mccauley@yahoo.fr being attributed to two different people ;)
  - justaguy37 5y ago
    can we just say no to capital letters? (or lowercase?) do capital letters have a good enough usage case to justify their continued existence?
    - colejohnson66 5y ago
      You are free to stop using capital letters, but good luck getting everyone to go along. Capitals have been around for centuries (they’re older than the printing press) and aren’t going anywhere.
      - bmn__ 5y ago
        The lower-case letters in Greek/Latin/Cyrillic are the new additions, initially we only had what is now called upper-case.
    - Lammy 5y ago
      Fun fact: The Apple Ⅱ and Ⅱ+ originally only did upper-case, and it was very popular to add a Shift Key / lower-case mod via one of the gamepad buttons: https://web.archive.org/web/20010212094858/http://home.swbell.net/rubywand/Csa2KBPADJS.html#001 https://web.archive.org/web/20010212094858/http://home.swbel...
  - vgel 5y ago
    First one: yes, though good UI should prevent it from happening unless the user really intended it (for example I have ~/Documents symlinked into Dropbox, so ~/documents could be local-only documents) Second one: no, emails are not filenames, and more generally distinguishability is more important for identifiers. In cases where identifiers like emails need to be mapped to filenames, like caches, they should be normalized.

arka2147483647 5y ago

That works for programmers, but not for users. There could be several files with the same name, buth with different encodings. Worse, depending on how your terminal encodes user input, some of them migth not be typable.

zarzavat 5y ago
From the users perspective I don't want any normalisation at all. It's good as long as you only have one file system but as soon as you get multiple file systems with conflicting rules (which includes transferring files to other people) it becomes hell. Unfortunately we are stuck with that hell.

tialaramex 5y ago

In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters. Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you could think of as UTF-16 code units but they don't promise to encode UTF-16. Probably the files have human readable names, but, maybe not. If you're accepting command line file names it's not crazy to insist on human readable (thus, Unicode) names, but if you process arbitrary input files you didn't create, particularly files you just found by looking around on disks unsupervised - you need to accept that utter gibberish is inevitable sooner or later and you must cope with that successfully. Rust's OSStr variants match this reality.

atoav 5y ago
This is what I found quite refreshing about Rust — instead of choosing one of the following: A) The programmer is a almighty god who knows everything, we just expose him to the raw thing B) The programmer is a immature toddler who cannot be trusted, so we handle things for them What Rust does is more among the lines of "you might already know this, but anyways here is a reminder that you, the programmer need to take some decision about this".
- deleted 5y ago
  [deleted]
GlitchMr 5y ago
Filenames in HFS+ filesystem (an old filesystem used by Mac OS X) are normalized with a proprietary variant of NFD - this is a filesystem feature. APFS removed this feature.
- 1over137 5y ago
  >APFS removed this feature. And then brought it back. It normalizes now.

stefan_ 5y ago

Sure but at some point you might want to create a file and frequently using user input or filter files using some user provided query string, the kind of use cases that unicode normalization was invented for. So the whole "opaque blob of bytes" filesystem handling is nice if all you want is to not silently corrupt files, but it is very obviously not even covering 10% of normal use cases. Rust isn't being super smart, it just has its hands thrown up in the air.

alkonaut 5y ago

Falls over on the fact that I don’t want to be able to write these two files in the same dir. if I write file ö1.txt and ö1.txt then I want to be warned that the file exists even of the encoding is different when I use two different apps but try to write the same file. The same applies for a.txt and A.txt on case insensitive file systems (as someone pointed out the most common desktop file systems are).

jrochkind1 5y ago

Well, precisely because if you don't normalize the filenames, ö ≠ ö. You could have two files with different filenames, `göteborg.txt` and `göteborg.txt`, and they are different files with different filenames. Or you could have one file `göteborg.txt`, and when you try to ask for it as `göteborg.txt`, the system tells you "no file by that name". Unicode normalization is the solution to this. And the unicode normalization algorithms are pretty good. The bug in this case is that the system did not apply unicode normalization consistently. It required a non-default config option to be turned on to do so? I don't really understand what's going on here, but it sounds like a bug in the system to me that this would be a non-default config option. Dealing with the entire universe of human language is inherently complicated. But unicode gives us some actually pretty marvelous tools for doing it consistently and reasonably. But you still have to use them, and use them right, and with all software bugs are possible. But I don't think you get fewer crazy edge cases by not normalizing at all. (In some cases you can even get security concerns, think about usernames and the risk of `jöhn` and `jöhn` being two different users...). I know that this is the choice some traditional/legacy OSs/file systems make, in order to keep pre-unicode-hegemony backwards compat. It has problems as well. I think the right choice for any greenfield possibilities is consistent unicode normalization, so `göteborg.txt` and `göteborg.txt` can't be two different files with two different filenames. [btw I tried to actually use the two common different forms of ö in this text; I don't believe HN normalizes them so they should remain.]

nieve 5y ago
It looks like instead of the config option switching everything to use the same normalization it keeps a second copy of the name in a database to compare to. What a horrible kludge, I wonder how they even got into this situation of using different normalization in different parts of the system?
- jrochkind1 5y ago
  That seems an odd choice indeed, because even if you do have different normalizations in differnet parts of the system, you don't need to keep multiple copies -- you just need to apply the right normalization in the right place. All of the unicode normalization algorithms are both idempotent and of course completely deterministic. If you apply NFD to any legal input, you get the same thing every time -- there's no need to store the NFC version separately to compare it to NFC input when all you have is NFD otherwise, you can just normalize the input to NFD to compare it to what you have! Unless it was meant to be for performance?