3 ms·

Indeed, this is exactly what the Linux kernel does; it treats filenames like a binary blob, with the exception that the bytes 0x2f (ASCII '/') and 0x00 (ASCII N

Indeed, this is exactly what the Linux kernel does; it treats filenames like a binary blob, with the exception that the bytes 0x2f (ASCII '/') and 0x00 (ASCII NUL) are not accepted (regardless of where they are in the byte string).

Kristine1975 10y ago
> it treats filenames like a binary blob Which can result in files with seemingly identical names when one UTF-8-encoded file name uses combining characters and the other one uses precomposed ones. Not to mention the fact that e.g. Å is contained in Unicode as both "latin capital letter A with ring above" and "Angstrom sign". After you've solved encoding, next comes Unicode normalization ;-)
- aleh 10y ago
  It is not a big deal as long as seemingly identical file names are treated by OS as different. Unicode normalization is not the only problem here, e.g. Latin 'a', 'e', 'T' are exactly the same as Cyrillic 'а', 'е', 'Т' in most fonts which makes it possible for two files to have seemingly same names even in some 8-bit encodings.
  - Hondor 10y ago
    Even Latin 'I' and 'l' and the digit '1' are visually indistinguishable in some fonts! So are trailing spaces and different numbers of spaces. This is such a pervasive problem that maybe we can just give up and expect users to get used to it.
Hondor 10y ago
Wow, so characters like U+022F (ȯ) and U+042F (Cyrillic letter Я) U+062F (Arabic letter د) are not allowed but nearly everything else is? Some of those are letters used in actual languages. That's sure to make people scratch their heads.
- pzb 10y ago
  When encoding those characters in UTF-8, you will never end up with 0x2F as a byte. On of the properties of UTF-8 is that bytes with the high bit not set (e.g. 0x0 to 0x7e) never appear unless the are representing the 0 - 127 codepoints.
- rdancer 10y ago
  codepoint -> encoding in UTF-8 U+022F -> C8 AF U+042F -> D0 AF U+062F -> D8 AF Remember, UTF-8 is self-synchronising: when you pick up at a random point within a stream, there is no ambiguity as to whether you are in the middle of a sequence or not. Valid lower codepoints appearing in the encoding of higher codepoints would violate this property.
- cataphract 10y ago
  If you encode them in UTF-8, they are allowed. In UTF-8, to represent a given code point, only the first code unit can be in the range 0x00 to 0x7F. So if you're reading UTF-8 content and find a byte with value 0x00-0x7F, you can be sure it is a 1-byte long code unit sequence representing the code points U+0000 to U+007F.

Kristine1975 10y ago

> it treats filenames like a binary blob Which can result in files with seemingly identical names when one UTF-8-encoded file name uses combining characters and the other one uses precomposed ones. Not to mention the fact that e.g. Å is contained in Unicode as both "latin capital letter A with ring above" and "Angstrom sign". After you've solved encoding, next comes Unicode normalization ;-)

aleh 10y ago
It is not a big deal as long as seemingly identical file names are treated by OS as different. Unicode normalization is not the only problem here, e.g. Latin 'a', 'e', 'T' are exactly the same as Cyrillic 'а', 'е', 'Т' in most fonts which makes it possible for two files to have seemingly same names even in some 8-bit encodings.
- Hondor 10y ago
  Even Latin 'I' and 'l' and the digit '1' are visually indistinguishable in some fonts! So are trailing spaces and different numbers of spaces. This is such a pervasive problem that maybe we can just give up and expect users to get used to it.

Hondor 10y ago

Wow, so characters like U+022F (ȯ) and U+042F (Cyrillic letter Я) U+062F (Arabic letter د) are not allowed but nearly everything else is? Some of those are letters used in actual languages. That's sure to make people scratch their heads.

pzb 10y ago
When encoding those characters in UTF-8, you will never end up with 0x2F as a byte. On of the properties of UTF-8 is that bytes with the high bit not set (e.g. 0x0 to 0x7e) never appear unless the are representing the 0 - 127 codepoints.
rdancer 10y ago
codepoint -> encoding in UTF-8 U+022F -> C8 AF U+042F -> D0 AF U+062F -> D8 AF Remember, UTF-8 is self-synchronising: when you pick up at a random point within a stream, there is no ambiguity as to whether you are in the middle of a sequence or not. Valid lower codepoints appearing in the encoding of higher codepoints would violate this property.
cataphract 10y ago
If you encode them in UTF-8, they are allowed. In UTF-8, to represent a given code point, only the first code unit can be in the range 0x00 to 0x7F. So if you're reading UTF-8 content and find a byte with value 0x00-0x7F, you can be sure it is a 1-byte long code unit sequence representing the code points U+0000 to U+007F.