4 ms·

Without some specific way of knowing what encoding you're dealing with, you can have this problem with any number of encodings you're working with (including AS

Without some specific way of knowing what encoding you're dealing with, you can have this problem with any number of encodings you're working with (including ASCII suddenly having UTF-8 bytes in it). As soon as you're in a "assume an encoding and maybe be surprised", you're asking to be surprised. Totally agree these discussions circle around to "you need some kind of OOB way to know what encoding you're dealing with", which puts us back to everything being encoding agnostic, with some way to configure it. Maybe that's locales, maybe that's a type 32-byte integer at the beginning, dunno.

kevin_thibedeau 5y ago
> including ASCII suddenly having UTF-8 bytes People have been dealing with 8-bit ASCII extensions for longer than Unicode has been around. It isn't overwhelmingly difficult to have 8-bit clean text handling without any awareness of the encoding.
jcranmer 5y ago
No, the best way is not to have people specify what encoding they intend to use--it is instead to not give people a choice. When you give people a choice, given that one encoding is far dominant above all others, many people will start assuming that is the only viable choice regardless of any actual declarations to the contrary. An example of the latter effect in action is the system charset of Linux and most other Unixen... in practice, virtually every system is using UTF-8 for things like pathnames and the like. However, the mechanism to indicate that this is the case is frequently enough unreliable that it's often less bug-prone to assume the system is UTF-8 anyways. Of course, there's a few more advantages to always using UTF-8: * You eliminate the code path of "I'm sorry, I can't save the text that you wrote," since every character is usable in UTF-8, which is a feature very rare outside of UTF-* charsets. And, honestly, that's a wonderful code path to eliminate. * One of the features of UTF-8 is that virtually any text that isn't UTF-8 will fail to parse as UTF-8. This allows you to build systems that produce errors on bad input, which means that maybe, just maybe, the people who produced the broken input will discover their problem and fix it.
- camgunz 5y ago
  > in practice, virtually every system is using UTF-8 for things like pathnames and the like. I mean, *nix filenames are a pretty weird case where you have these requirements of 0x2F always being a path separator and 0x00 always meaning "end of path". In truth, we can't guarantee anything about filename encoding other than those two things. So, for example, you might be dealing with UTF-8 or EUC-JP. In practice (as a sibling comment pointed out) all you usually need to do is scan for 0x2F and stop at 0x00, so this works out OK, but if you ever need to do anything else, welcome to guessing the encoding based on locales, environment variables, or other (OOB) configurations. > However, the mechanism to indicate that this is the case is frequently enough unreliable that it's often less bug-prone to assume the system is UTF-8 anyways. My argument, which I guess looking at it I've not articulated at all in this thread, is that UTF-8's ASCII compatibility and *nix filename compatibility have allowed us to be slack in fixing this. It oughta be possible to configure an encoding when creating a filesystem, just like creating a database. There's nothing inherently buggy about configuring an encoding, we just haven't gotten our systems programming act together, and UTF-8's amenability to the existing situation is probably at fault here. > since every character is usable in UTF-8 I guess you mean like, vs. EUC-JP or something? Sure but UTF-16 is fine here too. > One of the features of UTF-8 is that virtually any text that isn't UTF-8 will fail to parse as UTF-8. IDK the "brittleness" of an encoding feels hard to quantify. Besides, I think if you're doing the (always required) validation step, this goes for any input. Look at things like weirdo MySQL/Oracle/MSSQL database encodings for example. --- I should say I see where you're coming from. But I think we'd be in just as good a place if we specified a filename/text encoding at filesystem creation (OOB configuration) and quit relying on magic 0x2F/0x00 bytes. I don't know why this makes people tear their hair out as some kind of impossible problem to solve.

kevin_thibedeau 5y ago

> including ASCII suddenly having UTF-8 bytes People have been dealing with 8-bit ASCII extensions for longer than Unicode has been around. It isn't overwhelmingly difficult to have 8-bit clean text handling without any awareness of the encoding.

jcranmer 5y ago

No, the best way is not to have people specify what encoding they intend to use--it is instead to not give people a choice. When you give people a choice, given that one encoding is far dominant above all others, many people will start assuming that is the only viable choice regardless of any actual declarations to the contrary. An example of the latter effect in action is the system charset of Linux and most other Unixen... in practice, virtually every system is using UTF-8 for things like pathnames and the like. However, the mechanism to indicate that this is the case is frequently enough unreliable that it's often less bug-prone to assume the system is UTF-8 anyways. Of course, there's a few more advantages to always using UTF-8: * You eliminate the code path of "I'm sorry, I can't save the text that you wrote," since every character is usable in UTF-8, which is a feature very rare outside of UTF-* charsets. And, honestly, that's a wonderful code path to eliminate. * One of the features of UTF-8 is that virtually any text that isn't UTF-8 will fail to parse as UTF-8. This allows you to build systems that produce errors on bad input, which means that maybe, just maybe, the people who produced the broken input will discover their problem and fix it.

camgunz 5y ago
> in practice, virtually every system is using UTF-8 for things like pathnames and the like. I mean, *nix filenames are a pretty weird case where you have these requirements of 0x2F always being a path separator and 0x00 always meaning "end of path". In truth, we can't guarantee anything about filename encoding other than those two things. So, for example, you might be dealing with UTF-8 or EUC-JP. In practice (as a sibling comment pointed out) all you usually need to do is scan for 0x2F and stop at 0x00, so this works out OK, but if you ever need to do anything else, welcome to guessing the encoding based on locales, environment variables, or other (OOB) configurations. > However, the mechanism to indicate that this is the case is frequently enough unreliable that it's often less bug-prone to assume the system is UTF-8 anyways. My argument, which I guess looking at it I've not articulated at all in this thread, is that UTF-8's ASCII compatibility and *nix filename compatibility have allowed us to be slack in fixing this. It oughta be possible to configure an encoding when creating a filesystem, just like creating a database. There's nothing inherently buggy about configuring an encoding, we just haven't gotten our systems programming act together, and UTF-8's amenability to the existing situation is probably at fault here. > since every character is usable in UTF-8 I guess you mean like, vs. EUC-JP or something? Sure but UTF-16 is fine here too. > One of the features of UTF-8 is that virtually any text that isn't UTF-8 will fail to parse as UTF-8. IDK the "brittleness" of an encoding feels hard to quantify. Besides, I think if you're doing the (always required) validation step, this goes for any input. Look at things like weirdo MySQL/Oracle/MSSQL database encodings for example. --- I should say I see where you're coming from. But I think we'd be in just as good a place if we specified a filename/text encoding at filesystem creation (OOB configuration) and quit relying on magic 0x2F/0x00 bytes. I don't know why this makes people tear their hair out as some kind of impossible problem to solve.