3 ms·

Sure it's useful, but it's not at all a guarantee. You can't assume the byte stream is UTF-8, and UTF-8 has a NUL in it. Reading about it, filename encoding lo

Sure it's useful, but it's not at all a guarantee. You can't assume the byte stream is UTF-8, and UTF-8 has a NUL in it. Reading about it, filename encoding looks like a mess, GLib apparently assumes UTF-8 unless you set it with an environment variable, which I think is wild. Qt uses the locale, which feels reasonable (ignoring that locales are themselves bananas). But assuming UTF-8 and relying on ASCII compatibility and no NULs feels risky.

Dylan16807 5y ago
ASCII has NUL too. You can't put it in a filename either way.
- camgunz 5y ago
  Yeah but there's no reason to put it in strings because it's not printable. When you get to the 2/3/4 byte-wide parts or UTF-8 they can have zeroes in there. This is a long-winded way of saying UTF-8 can't really be NULL-terminated because it can contain NULLs as a matter of course. That's where you get modified UTF-8, which is indistinguishable from regular UTF-8 other than it's NULL encoding, which if you don't know it's the modified derivative will sneak up on you, just like UCS-2 vs UTF-16. This stuff just has to be OOB configurable. ASCII-compatibility lets us think we can do stuff like this, but you're always just assuming and hoping it doesn't blow up.
  - Dylan16807 5y ago
    > This is a long-winded way of saying UTF-8 can't really be NULL-terminated because it can contain NULLs as a matter of course. UTF-8 "can contain NULLs as a matter of course" in exactly the same way that ASCII can. Any system that doesn't allow null bytes in ASCII can just as easily not allow them in UTF-8. A system that does allow them can just as easily allow them in both. There is no difference between the two here. - Edit: Oh, wait, maybe you misunderstand UTF-8. The 2+ byte encodings never use bytes below 0x80. They never cause a null byte to happen. The only way you get a null byte is if it's an ASCII-compatible null character, code point 00000.
    - camgunz 5y ago
      Oop, sorry it's late here and I forgot that. OK, I'll grant you UTF-8 (or I guess any NULL-terminatable encoding like Shift JIS or EUC-JP) is better for filenames.
  - toast0 5y ago
    UTF-8 Doesn't contain zero bytes, except for code point zero. Anything above code point 127 will be encoded as multiple bytes, all of which have bit 7 (0x80) set and are therefore not a zero byte. So NULL-terminated works fine for utf-8.
  - deleted 5y ago
    [deleted]

Dylan16807 5y ago

ASCII has NUL too. You can't put it in a filename either way.

camgunz 5y ago
Yeah but there's no reason to put it in strings because it's not printable. When you get to the 2/3/4 byte-wide parts or UTF-8 they can have zeroes in there. This is a long-winded way of saying UTF-8 can't really be NULL-terminated because it can contain NULLs as a matter of course. That's where you get modified UTF-8, which is indistinguishable from regular UTF-8 other than it's NULL encoding, which if you don't know it's the modified derivative will sneak up on you, just like UCS-2 vs UTF-16. This stuff just has to be OOB configurable. ASCII-compatibility lets us think we can do stuff like this, but you're always just assuming and hoping it doesn't blow up.
- Dylan16807 5y ago
  > This is a long-winded way of saying UTF-8 can't really be NULL-terminated because it can contain NULLs as a matter of course. UTF-8 "can contain NULLs as a matter of course" in exactly the same way that ASCII can. Any system that doesn't allow null bytes in ASCII can just as easily not allow them in UTF-8. A system that does allow them can just as easily allow them in both. There is no difference between the two here. - Edit: Oh, wait, maybe you misunderstand UTF-8. The 2+ byte encodings never use bytes below 0x80. They never cause a null byte to happen. The only way you get a null byte is if it's an ASCII-compatible null character, code point 00000.
  - camgunz 5y ago
    Oop, sorry it's late here and I forgot that. OK, I'll grant you UTF-8 (or I guess any NULL-terminatable encoding like Shift JIS or EUC-JP) is better for filenames.
- toast0 5y ago
  UTF-8 Doesn't contain zero bytes, except for code point zero. Anything above code point 127 will be encoded as multiple bytes, all of which have bit 7 (0x80) set and are therefore not a zero byte. So NULL-terminated works fine for utf-8.
- deleted 5y ago
  [deleted]