Yeah I mean, I broadly agree with what you're saying. Text encoding is a mess, and pragmatic solutions are needed. And when jamming text in various places, jamm

Yeah I mean, I broadly agree with what you're saying. Text encoding is a mess, and pragmatic solutions are needed. And when jamming text in various places, jamming UTF-8 is typically more successful than jamming in UTF-16, because of NULL bytes and the shortsightedness of UCS-2. But I think that 99% of the things you have to do for UTF-8 (validate, encode/decode, no indexing) you have to do for UTF-16, and if we're arguing for systems that are UTF-aware, UTF-16 does great. If we're arguing for trying to jam encoded bytes into systems, well it's a crapshoot. UTF-8 being better at winning that crapshoot doesn't feel like a recipe for robustness to me, and certainly doesn't strike me as a perfect solution. OOB configuration does, though. > For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Well, this is a weird example. grep uses locales, so yeah you're probably fine if you're grepping for ASCII using an ASCII-compatible locale. But if you're not, then what you feed to grep has to be encoded in your locale's encoding (I think). So I think this is actually an example in favor of "this should be OOB configurable", and an example of "ASCII-compatibility lulls into a false sense of security".