5 ms·

Filenames come from multiple filesystems (and not just filesystems!), with different encodings and such. Both ntfs and ext3 can use arbitrary bytes as names, no

Filenames come from multiple filesystems (and not just filesystems!), with different encodings and such. Both ntfs and ext3 can use arbitrary bytes as names, not representable in utf8. (Contrary to popular belief, ntfs works with any byte sequences except 0x0000, and may contain invalid utf16 code points) The only sane way to deal with them all is to use bytes and bytes only (Or use a sane language with encoding-agnostic string type; see: golang) Other solutions introduce loss of data.

chungy 4y ago
NTFS disallows 0x002F as well; it is possible to use the entire set even from Windows (installing Interix, for instance). Windows NT was designed around UCS-2, the file system didn't have to do much but be a dumb bag of bytes for file names, character interpretation was on upper layers. You can mostly pretend it's UTF-16, until it's not. Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems.
- jcranmer 4y ago
  > Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems. This is the sort of option that should be available for all filesystems, and distros should default new installs to turning it on.
  - deckard1 4y ago
    if we have a time machine then yes. In fact, it shouldn't even be an option. UTF-8 period. Always. But alas, we do not and someone will plug in a memory card or USB drive with some FAT formatted drive and your nice universe crumbles before you.
    - chungy 4y ago
      This is why "convmv" exists :) and also why my /tmp is a normal tmpfs where file names can be any byte sequence. I can, for instance, extract random archives there, use convmv to make them UTF-8 (if not already), and then it can go on my ZFS file systems.
    - jcranmer 4y ago
      Handling non-UTF-8 pathnames via a filesystem mount option is completely viable. Hell, I'd go even further and suggest that the kernel itself reject non-UTF-8 pathnames entirely, handling filesystems with the non-UTF-8-pathname option enabled via a translation layer that converts \x00-\xff to \u0000-\u00ff (and vice versa). The lesson of text is clear: text needs to have a defined charset. Filesystem names are absolutely textual (listing a directory for human display is one of the most common operations). If it's not UTF-8, then what is it? How is the application supposed to figure it out? (Automatic encoding detection is flaky in the best of times, but with the lengths of typical filenames, it is almost impossible to be reliable). The reality is that most applications already assume that filesystem names are UTF-8, and non-UTF-8 names break them. We are at a stage in Unicode penetration that I think it is reasonable to treat a filesystem that has non-UTF-8 filenames as a problem that needs to be solved with some kind of repair tool like fsck, not something that every application is supposed to worry about.
  - lmm 4y ago
    Please don't unless you've thoroughly tested in multiple cultures. Unicode is broken for Japanese; on some systems it's possible to change a setting to make it broken for Chinese instead, but the only non-theoretical way to make a program that can list a directory full of Japanese filenames and a directory full of Chinese filenames correctly is to have that program be encoding-aware.
    - lidavidm 4y ago
      I'm curious, what does 'broken' mean? Is this about the CJK merger, or something else?
      - lmm 4y ago
        Yes, CJK unification. Characters will be rendered differently from how they should be (e.g. with missing or extra strokes).
        hnfong 4y ago
        Why is that a filesystem issue or a "utf8only=on" issue? If the renderer doesn't have sufficient context, it will pick one glyph to render anyway, and it could be "wrong". PS: unless your take is to say that we should just assume everything is ascii and don't ever render codepoints > 127?
        lmm 4y ago
        If the filesystem can't allow parts to be in different encodings then it can't ever possibly work right. With current filesystems at least I can do "ls ~/chinese-files; LANG=ja_JP.SJIS ls ~/japanese-files". Which is not a great way of working, but it's better than a utf8only filesystem where it's physically impossible to make even that much work.
        chungy 4y ago
        If you run a ZFS file system where this is a huge deal to you, then don't set utf8only. If it's set on a parent dataset, then explicitly do utf8only=off.
deathanatos 4y ago
> The only sane way to deal with them all is to use bytes and bytes only It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it, you're back into the world of hurt the article describes, and you need some super complicated type or logic that can convey to the user "hey, this file's name is some random undecodable garbage". The sane state would have been "file names are text", but filenames/Unix predates software engineering having a good discipline around text encoding, and it predates Unicode. I'd go as far as to argue that "file names are text and \n is disallowed" is the sane state, as multi-line filenames also just cause all sorts of unnecessary complexity. The problem, of course, is that most OS enforce neither constraint, which is what leads to things like the struggle Python has gone through, or the Path type in Rust, and the unrealistic nature of trying to move from the present state to such a sane state. (Linux is loathe to break compatibility, and I bet Windows would be similar, here. There are a number of arguments for that, but in the meantime, it continues to be that file names are ridiculously complicated if handled fully correctly. Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.) > Or use a sane language with encoding-agnostic string type; see: golang Of course, this might be the root of our disagreement. Types that permit the representation of invalid values are not desirable, to me. Especially in a core type, like strings.
- saurik 4y ago
  > Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.) You have to be really careful about the way in which you give up here, as the user isn't necessarily in control of the files on disk: it might be an attacker. If I can hide files in folders from various parts of a Python program -- as was possible with early versions of Python 3, which simply skipped such files -- or cause corruption in how the filenames are round-tripped during a copy (allowing me to bypass a filter in one place but then write to a file with a different critical name in another place) I gain a lot of flexibility with my exploits.
  - 4y ago
russdill 4y ago
Even using bytes can lead to loss of data when trying to copy from one file-system to another or dealing with operating systems that treat filenames differently when different system calls are used.
gumby 4y ago
I have used filesystems that permitted a character represented as 0x0. Good way to attack a C program.
- GeorgeTirebiter 4y ago
  I sometimes think the C decision to encode strings as non-zero bytes was one of the worst. The other candidate is the atrocious operator precedence. OK, we now love parens to 'fix' this -- might as well use lisp... ;-)
  - ilyt 4y ago
    It was unequivocally one of the worst decisions in programming. I wouldn't be surprised if costs of dealing with it would be in the billions now. It occasionally saved a byte on devices with little memory which kinda made sense at a time, but not anymore
    - stabbles 4y ago
      It makes perfect sense when writing and reading strings from disk, since you don’t have to worry where and how to store the integer for the length (#bytes & endianness)
      - optionalsquid 4y ago
        Wouldn't endianness be a concern regardless, unless you are literally only reading/writing (null terminated) strings? Which I would imagine to be a much rarer case than writing a mix of data-types or just plain text.
  - gumby 4y ago
    Many systems used this convention; c was merely one of them. So it was a “choice” more than an”decision”. The Mesa guys at PARC did a study back in the late 70s and, statistically, length-marked strings are faster than delimited ones. Delimited strings have some advantages: they save the space needed to store the length, at the cost of an extra byte (not worth the cost these days); and some substrings can be assembled simply by pointing into the string (of course counted strings can share storage). But at a terrible cost. But remember that the PDP-7 was less powerful than some AVRs. We use more powerful chips to scan keyboards these days.

chungy 4y ago

NTFS disallows 0x002F as well; it is possible to use the entire set even from Windows (installing Interix, for instance). Windows NT was designed around UCS-2, the file system didn't have to do much but be a dumb bag of bytes for file names, character interpretation was on upper layers. You can mostly pretend it's UTF-16, until it's not. Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems.

jcranmer 4y ago
> Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems. This is the sort of option that should be available for all filesystems, and distros should default new installs to turning it on.
- deckard1 4y ago
  if we have a time machine then yes. In fact, it shouldn't even be an option. UTF-8 period. Always. But alas, we do not and someone will plug in a memory card or USB drive with some FAT formatted drive and your nice universe crumbles before you.
  - chungy 4y ago
    This is why "convmv" exists :) and also why my /tmp is a normal tmpfs where file names can be any byte sequence. I can, for instance, extract random archives there, use convmv to make them UTF-8 (if not already), and then it can go on my ZFS file systems.
  - jcranmer 4y ago
    Handling non-UTF-8 pathnames via a filesystem mount option is completely viable. Hell, I'd go even further and suggest that the kernel itself reject non-UTF-8 pathnames entirely, handling filesystems with the non-UTF-8-pathname option enabled via a translation layer that converts \x00-\xff to \u0000-\u00ff (and vice versa). The lesson of text is clear: text needs to have a defined charset. Filesystem names are absolutely textual (listing a directory for human display is one of the most common operations). If it's not UTF-8, then what is it? How is the application supposed to figure it out? (Automatic encoding detection is flaky in the best of times, but with the lengths of typical filenames, it is almost impossible to be reliable). The reality is that most applications already assume that filesystem names are UTF-8, and non-UTF-8 names break them. We are at a stage in Unicode penetration that I think it is reasonable to treat a filesystem that has non-UTF-8 filenames as a problem that needs to be solved with some kind of repair tool like fsck, not something that every application is supposed to worry about.
- lmm 4y ago
  Please don't unless you've thoroughly tested in multiple cultures. Unicode is broken for Japanese; on some systems it's possible to change a setting to make it broken for Chinese instead, but the only non-theoretical way to make a program that can list a directory full of Japanese filenames and a directory full of Chinese filenames correctly is to have that program be encoding-aware.
  - lidavidm 4y ago
    I'm curious, what does 'broken' mean? Is this about the CJK merger, or something else?
    - lmm 4y ago
      Yes, CJK unification. Characters will be rendered differently from how they should be (e.g. with missing or extra strokes).
      - hnfong 4y ago
        Why is that a filesystem issue or a "utf8only=on" issue? If the renderer doesn't have sufficient context, it will pick one glyph to render anyway, and it could be "wrong". PS: unless your take is to say that we should just assume everything is ascii and don't ever render codepoints > 127?
        lmm 4y ago
        If the filesystem can't allow parts to be in different encodings then it can't ever possibly work right. With current filesystems at least I can do "ls ~/chinese-files; LANG=ja_JP.SJIS ls ~/japanese-files". Which is not a great way of working, but it's better than a utf8only filesystem where it's physically impossible to make even that much work.
        chungy 4y ago
        If you run a ZFS file system where this is a huge deal to you, then don't set utf8only. If it's set on a parent dataset, then explicitly do utf8only=off.

deathanatos 4y ago

> The only sane way to deal with them all is to use bytes and bytes only It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it, you're back into the world of hurt the article describes, and you need some super complicated type or logic that can convey to the user "hey, this file's name is some random undecodable garbage". The sane state would have been "file names are text", but filenames/Unix predates software engineering having a good discipline around text encoding, and it predates Unicode. I'd go as far as to argue that "file names are text and \n is disallowed" is the sane state, as multi-line filenames also just cause all sorts of unnecessary complexity. The problem, of course, is that most OS enforce neither constraint, which is what leads to things like the struggle Python has gone through, or the Path type in Rust, and the unrealistic nature of trying to move from the present state to such a sane state. (Linux is loathe to break compatibility, and I bet Windows would be similar, here. There are a number of arguments for that, but in the meantime, it continues to be that file names are ridiculously complicated if handled fully correctly. Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.) > Or use a sane language with encoding-agnostic string type; see: golang Of course, this might be the root of our disagreement. Types that permit the representation of invalid values are not desirable, to me. Especially in a core type, like strings.

saurik 4y ago
> Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.) You have to be really careful about the way in which you give up here, as the user isn't necessarily in control of the files on disk: it might be an attacker. If I can hide files in folders from various parts of a Python program -- as was possible with early versions of Python 3, which simply skipped such files -- or cause corruption in how the filenames are round-tripped during a copy (allowing me to bypass a filter in one place but then write to a file with a different critical name in another place) I gain a lot of flexibility with my exploits.
- 4y ago

russdill 4y ago

Even using bytes can lead to loss of data when trying to copy from one file-system to another or dealing with operating systems that treat filenames differently when different system calls are used.

gumby 4y ago

I have used filesystems that permitted a character represented as 0x0. Good way to attack a C program.

GeorgeTirebiter 4y ago
I sometimes think the C decision to encode strings as non-zero bytes was one of the worst. The other candidate is the atrocious operator precedence. OK, we now love parens to 'fix' this -- might as well use lisp... ;-)
- ilyt 4y ago
  It was unequivocally one of the worst decisions in programming. I wouldn't be surprised if costs of dealing with it would be in the billions now. It occasionally saved a byte on devices with little memory which kinda made sense at a time, but not anymore
  - stabbles 4y ago
    It makes perfect sense when writing and reading strings from disk, since you don’t have to worry where and how to store the integer for the length (#bytes & endianness)
    - optionalsquid 4y ago
      Wouldn't endianness be a concern regardless, unless you are literally only reading/writing (null terminated) strings? Which I would imagine to be a much rarer case than writing a mix of data-types or just plain text.
- gumby 4y ago
  Many systems used this convention; c was merely one of them. So it was a “choice” more than an”decision”. The Mesa guys at PARC did a study back in the late 70s and, statistically, length-marked strings are faster than delimited ones. Delimited strings have some advantages: they save the space needed to store the length, at the cost of an extra byte (not worth the cost these days); and some substrings can be assembled simply by pointing into the string (of course counted strings can share storage). But at a terrible cost. But remember that the PDP-7 was less powerful than some AVRs. We use more powerful chips to scan keyboards these days.