5 ms·

Filenames with Accents (2011)

21 points by FrankSansC 4y ago

juancn 4y ago
You should normalize names on write, on read is very hard to fix. You can have a perfectly valid, denormalized strings representing codepoints with different normalizations. So if you have four possible normalizations: NFD, NFC, NFKD, NFKC and your string has N ambiguous codepoints, the number of possible strings you need to try is N^4.
- asddubs 4y ago
  seems like the only way to really achieve that is to bake it into the api for storing/naming files of the respective operating system
  - enriquto 4y ago
    Indeed. The unix world would be a much happier place if the creat system call normalized the strings it receives to replace literal spaces with non-breaking spaces, and similar stuff. Regular users wouldn't notice, and it would simplify tons of shell scripts.
    - Someone 4y ago
      > if the creat system call normalized the strings it receives Nowadays, there’s an understanding to assume those bytes are strings encoded in some ISO-8859 variant or UTF-8, but technically, the creat system call doesn’t receive strings; it receives byte arrays. Historically, that was the (somewhat) right decision because it meant file systems didn’t need to know much about character encodings (they only needed to know the byte value of ‘/‘ and that zero is the name terminator), giving you a nice separation of concerns. With Unicode, if you want to normalize names on write, or even only reject incorrectly normalized names, or have case-insensitive file names, your file system code needs to know a lot of Unicode. That can be problematic on small embedded systems.
      - asddubs 4y ago
        I guess they could make it a compile flag
        Someone 4y ago
        And have tool 1 happily create files that, according to tool 2, compiled with a different flag setting, cannot exist or create multiple files in a directory that, according to tool 2 have the same names? I know two sort-of examples of this. Firstly, there’s MS-DOS long file names. That’s a hack (in the positive sense of the word) that gives code that doesn’t know about long file names the 8.3 file names that it expects. It works, but code that isn’t aware of long file names will only write 8.3 ones, so even a simple file copy using an old copy tool will drop long file names. Secondly, there’s macOS. It has a minor thing with directory separators. The Unix layer thinks ‘/‘ is the directory separator, old Mac code thinks it’s ‘:’. That works relatively well, mostly because old style Mac code doesn’t use file paths in the UI. And of course, every desktop OS has to deal with it when it mounts various drives that handle file names differently. HFS+ used NFD normalization, most other file systems that normalize names use NFC, disks may be case preserving or not, case insensitive or not, normalization insensitive or not, etc. On Unix-like OSes a path /a/b/c/d/e/f can walk six different file systems, each with different rules (even without soft or hard links). It wouldn’t surprise me to find bugs in programs there.
        asddubs 4y ago
        well, you'd just be in the same situation we're already in: you might not find the file by typing the name. you'd have to support files created before this flag was toggled on anyway, so it's not like everyone would just start assuming all files follow the convention
- rurban 4y ago
  only use NFC. NFD was used by HFS+, but got abandandoned by the current insecure APFS. (which uses unidentifiable names). NFD is faster to produce, but NFC is complete and needs less space. with NFD you can still have reordered sequence variants, and thus nonidentifiable names. and the NFKD, NFKC hacks should only be used with python internally, because they didn't understood Unicode. or just read the TR's without understanding it. it will need several more decades until filesystems will find out about their wrong decisions. maybe I'll bug them with CVE's some day.
baal80spam 4y ago
Side note - after all these years I still don't feel comfortable with using special characters (like ą, ż, ź) and spaces in filenames in Windows. DOS times sit deeply in my soul and It just doesn't feel right.
- klyrs 4y ago
  I've learned to live without file extensions... but I still wince at 4-letter file extensions. I haven't used windows in almost 20 years.
  - Tagbert 4y ago
    Do your files really not have file extensions or is it just that they are hidden by default in the file browser? In Mac OS extensions are important but they are hidden by default. I always force them to be shown when I setup a new Mac. Hiding them obscures important information. Hiding them offers nearly no benefit.
    - klyrs 4y ago
      I use Linux exclusively, where file extensions are entirely optional. While, yes, they are shown in the file browser, that's a tool of last resort for me: my display is typically a grid of terminals and I do most of my work through them. Text files, such as executable scripts with shebangs, do not get file extensions. Plain text files, such as when I'm taking notes about something, do not get file extensions. Stuff like images, music, etc get file extensions.
- renewiltord 4y ago
  This is amusing but my filesystem has no spaces in it. It's still a pain in the ass from the command line to handle spaces and instead of `-print0` and all that shit I just ban spaces from my filesystem. `-` and `_` are sufficient as spacers. I know zsh handles auto-complete well but I can `y-a-w` out of my `nvim` `:terminal` a lot easier.
  - 4y ago
Anthony-G 4y ago
In 2009, David A. Wheeler wrote a comprehensive article covering problems with Unix/Linux/POSIX filenames¹. Given that the OS naïvely treats filenames as a simple stream of bytes, he advocated that developers use UTF-8 for encoding filenames. He mentioned the issue of multiple normalisation systems being used to encode characters that have more than one Unicode representation but glossed over it because such problems are “overshadowed by the terrible awful even worse problems caused by filenames all being in random unguessable charsets”. I’m guessing that, by now, most developers on Unix-like systems would be using UTF-8 for filenames – though a decade after these articles were published, there still doesn’t seem to be any good/universal solution to the problem of characters with multiple Unicode representations. ¹ https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html

21 points by FrankSansC 4y ago

You should normalize names on write, on read is very hard to fix. You can have a perfectly valid, denormalized strings representing codepoints with different normalizations. So if you have four possible normalizations: NFD, NFC, NFKD, NFKC and your string has N ambiguous codepoints, the number of possible strings you need to try is N^4.

asddubs 4y ago
seems like the only way to really achieve that is to bake it into the api for storing/naming files of the respective operating system
- enriquto 4y ago
  Indeed. The unix world would be a much happier place if the creat system call normalized the strings it receives to replace literal spaces with non-breaking spaces, and similar stuff. Regular users wouldn't notice, and it would simplify tons of shell scripts.
  - Someone 4y ago
    > if the creat system call normalized the strings it receives Nowadays, there’s an understanding to assume those bytes are strings encoded in some ISO-8859 variant or UTF-8, but technically, the creat system call doesn’t receive strings; it receives byte arrays. Historically, that was the (somewhat) right decision because it meant file systems didn’t need to know much about character encodings (they only needed to know the byte value of ‘/‘ and that zero is the name terminator), giving you a nice separation of concerns. With Unicode, if you want to normalize names on write, or even only reject incorrectly normalized names, or have case-insensitive file names, your file system code needs to know a lot of Unicode. That can be problematic on small embedded systems.
    - asddubs 4y ago
      I guess they could make it a compile flag
      - Someone 4y ago
        And have tool 1 happily create files that, according to tool 2, compiled with a different flag setting, cannot exist or create multiple files in a directory that, according to tool 2 have the same names? I know two sort-of examples of this. Firstly, there’s MS-DOS long file names. That’s a hack (in the positive sense of the word) that gives code that doesn’t know about long file names the 8.3 file names that it expects. It works, but code that isn’t aware of long file names will only write 8.3 ones, so even a simple file copy using an old copy tool will drop long file names. Secondly, there’s macOS. It has a minor thing with directory separators. The Unix layer thinks ‘/‘ is the directory separator, old Mac code thinks it’s ‘:’. That works relatively well, mostly because old style Mac code doesn’t use file paths in the UI. And of course, every desktop OS has to deal with it when it mounts various drives that handle file names differently. HFS+ used NFD normalization, most other file systems that normalize names use NFC, disks may be case preserving or not, case insensitive or not, normalization insensitive or not, etc. On Unix-like OSes a path /a/b/c/d/e/f can walk six different file systems, each with different rules (even without soft or hard links). It wouldn’t surprise me to find bugs in programs there.
        asddubs 4y ago
        well, you'd just be in the same situation we're already in: you might not find the file by typing the name. you'd have to support files created before this flag was toggled on anyway, so it's not like everyone would just start assuming all files follow the convention
rurban 4y ago
only use NFC. NFD was used by HFS+, but got abandandoned by the current insecure APFS. (which uses unidentifiable names). NFD is faster to produce, but NFC is complete and needs less space. with NFD you can still have reordered sequence variants, and thus nonidentifiable names. and the NFKD, NFKC hacks should only be used with python internally, because they didn't understood Unicode. or just read the TR's without understanding it. it will need several more decades until filesystems will find out about their wrong decisions. maybe I'll bug them with CVE's some day.

baal80spam 4y ago

Side note - after all these years I still don't feel comfortable with using special characters (like ą, ż, ź) and spaces in filenames in Windows. DOS times sit deeply in my soul and It just doesn't feel right.

klyrs 4y ago
I've learned to live without file extensions... but I still wince at 4-letter file extensions. I haven't used windows in almost 20 years.
- Tagbert 4y ago
  Do your files really not have file extensions or is it just that they are hidden by default in the file browser? In Mac OS extensions are important but they are hidden by default. I always force them to be shown when I setup a new Mac. Hiding them obscures important information. Hiding them offers nearly no benefit.
  - klyrs 4y ago
    I use Linux exclusively, where file extensions are entirely optional. While, yes, they are shown in the file browser, that's a tool of last resort for me: my display is typically a grid of terminals and I do most of my work through them. Text files, such as executable scripts with shebangs, do not get file extensions. Plain text files, such as when I'm taking notes about something, do not get file extensions. Stuff like images, music, etc get file extensions.
renewiltord 4y ago
This is amusing but my filesystem has no spaces in it. It's still a pain in the ass from the command line to handle spaces and instead of `-print0` and all that shit I just ban spaces from my filesystem. `-` and `_` are sufficient as spacers. I know zsh handles auto-complete well but I can `y-a-w` out of my `nvim` `:terminal` a lot easier.
- 4y ago

Anthony-G 4y ago

In 2009, David A. Wheeler wrote a comprehensive article covering problems with Unix/Linux/POSIX filenames¹. Given that the OS naïvely treats filenames as a simple stream of bytes, he advocated that developers use UTF-8 for encoding filenames. He mentioned the issue of multiple normalisation systems being used to encode characters that have more than one Unicode representation but glossed over it because such problems are “overshadowed by the terrible awful even worse problems caused by filenames all being in random unguessable charsets”. I’m guessing that, by now, most developers on Unix-like systems would be using UTF-8 for filenames – though a decade after these articles were published, there still doesn’t seem to be any good/universal solution to the problem of characters with multiple Unicode representations. ¹ https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html