3 ms·

NTFS disallows 0x002F as well; it is possible to use the entire set even from Windows (installing Interix, for instance). Windows NT was designed around UCS-2,

by chungy 4y ago

NTFS disallows 0x002F as well; it is possible to use the entire set even from Windows (installing Interix, for instance). Windows NT was designed around UCS-2, the file system didn't have to do much but be a dumb bag of bytes for file names, character interpretation was on upper layers. You can mostly pretend it's UTF-16, until it's not. Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems.

jcranmer 4y ago
> Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems. This is the sort of option that should be available for all filesystems, and distros should default new installs to turning it on.
- deckard1 4y ago
  if we have a time machine then yes. In fact, it shouldn't even be an option. UTF-8 period. Always. But alas, we do not and someone will plug in a memory card or USB drive with some FAT formatted drive and your nice universe crumbles before you.
  - chungy 4y ago
    This is why "convmv" exists :) and also why my /tmp is a normal tmpfs where file names can be any byte sequence. I can, for instance, extract random archives there, use convmv to make them UTF-8 (if not already), and then it can go on my ZFS file systems.
  - jcranmer 4y ago
    Handling non-UTF-8 pathnames via a filesystem mount option is completely viable. Hell, I'd go even further and suggest that the kernel itself reject non-UTF-8 pathnames entirely, handling filesystems with the non-UTF-8-pathname option enabled via a translation layer that converts \x00-\xff to \u0000-\u00ff (and vice versa). The lesson of text is clear: text needs to have a defined charset. Filesystem names are absolutely textual (listing a directory for human display is one of the most common operations). If it's not UTF-8, then what is it? How is the application supposed to figure it out? (Automatic encoding detection is flaky in the best of times, but with the lengths of typical filenames, it is almost impossible to be reliable). The reality is that most applications already assume that filesystem names are UTF-8, and non-UTF-8 names break them. We are at a stage in Unicode penetration that I think it is reasonable to treat a filesystem that has non-UTF-8 filenames as a problem that needs to be solved with some kind of repair tool like fsck, not something that every application is supposed to worry about.
- lmm 4y ago
  Please don't unless you've thoroughly tested in multiple cultures. Unicode is broken for Japanese; on some systems it's possible to change a setting to make it broken for Chinese instead, but the only non-theoretical way to make a program that can list a directory full of Japanese filenames and a directory full of Chinese filenames correctly is to have that program be encoding-aware.
  - lidavidm 4y ago
    I'm curious, what does 'broken' mean? Is this about the CJK merger, or something else?
    - lmm 4y ago
      Yes, CJK unification. Characters will be rendered differently from how they should be (e.g. with missing or extra strokes).
      - hnfong 4y ago
        Why is that a filesystem issue or a "utf8only=on" issue? If the renderer doesn't have sufficient context, it will pick one glyph to render anyway, and it could be "wrong". PS: unless your take is to say that we should just assume everything is ascii and don't ever render codepoints > 127?
        lmm 4y ago
        If the filesystem can't allow parts to be in different encodings then it can't ever possibly work right. With current filesystems at least I can do "ls ~/chinese-files; LANG=ja_JP.SJIS ls ~/japanese-files". Which is not a great way of working, but it's better than a utf8only filesystem where it's physically impossible to make even that much work.
        chungy 4y ago
        If you run a ZFS file system where this is a huge deal to you, then don't set utf8only. If it's set on a parent dataset, then explicitly do utf8only=off.

by chungy 4y ago

jcranmer 4y ago

> Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems. This is the sort of option that should be available for all filesystems, and distros should default new installs to turning it on.

deckard1 4y ago
if we have a time machine then yes. In fact, it shouldn't even be an option. UTF-8 period. Always. But alas, we do not and someone will plug in a memory card or USB drive with some FAT formatted drive and your nice universe crumbles before you.
- chungy 4y ago
  This is why "convmv" exists :) and also why my /tmp is a normal tmpfs where file names can be any byte sequence. I can, for instance, extract random archives there, use convmv to make them UTF-8 (if not already), and then it can go on my ZFS file systems.
- jcranmer 4y ago
  Handling non-UTF-8 pathnames via a filesystem mount option is completely viable. Hell, I'd go even further and suggest that the kernel itself reject non-UTF-8 pathnames entirely, handling filesystems with the non-UTF-8-pathname option enabled via a translation layer that converts \x00-\xff to \u0000-\u00ff (and vice versa). The lesson of text is clear: text needs to have a defined charset. Filesystem names are absolutely textual (listing a directory for human display is one of the most common operations). If it's not UTF-8, then what is it? How is the application supposed to figure it out? (Automatic encoding detection is flaky in the best of times, but with the lengths of typical filenames, it is almost impossible to be reliable). The reality is that most applications already assume that filesystem names are UTF-8, and non-UTF-8 names break them. We are at a stage in Unicode penetration that I think it is reasonable to treat a filesystem that has non-UTF-8 filenames as a problem that needs to be solved with some kind of repair tool like fsck, not something that every application is supposed to worry about.
lmm 4y ago
Please don't unless you've thoroughly tested in multiple cultures. Unicode is broken for Japanese; on some systems it's possible to change a setting to make it broken for Chinese instead, but the only non-theoretical way to make a program that can list a directory full of Japanese filenames and a directory full of Chinese filenames correctly is to have that program be encoding-aware.
- lidavidm 4y ago
  I'm curious, what does 'broken' mean? Is this about the CJK merger, or something else?
  - lmm 4y ago
    Yes, CJK unification. Characters will be rendered differently from how they should be (e.g. with missing or extra strokes).
    - hnfong 4y ago
      Why is that a filesystem issue or a "utf8only=on" issue? If the renderer doesn't have sufficient context, it will pick one glyph to render anyway, and it could be "wrong". PS: unless your take is to say that we should just assume everything is ascii and don't ever render codepoints > 127?
      - lmm 4y ago
        If the filesystem can't allow parts to be in different encodings then it can't ever possibly work right. With current filesystems at least I can do "ls ~/chinese-files; LANG=ja_JP.SJIS ls ~/japanese-files". Which is not a great way of working, but it's better than a utf8only filesystem where it's physically impossible to make even that much work.
        chungy 4y ago
        If you run a ZFS file system where this is a huge deal to you, then don't set utf8only. If it's set on a parent dataset, then explicitly do utf8only=off.