3 ms·
> The way I see it, Unicode is only relevant for displaying strings to humans, or for taking input from humans directly. For virtually all other purposes, strin
by zzo38computer 2y ago
> The way I see it, Unicode is only relevant for displaying strings to humans, or for taking input from humans directly. For virtually all other purposes, strings should be treated as byte sequences internally, regardless of whether they were intended to be UTF-8 or something else. For example, if you're reading a JSON document and looking for a hardcoded key, there's no reason whatsoever to represent the JSON or the key as Unicode. The key is a sequence of bytes, the JSON objects have sequences of bytes as keys. The fact that JSON usually prefers UTF-8 is of relatively little relevance.
This is true (although Unicode is not the best character set, but that is a separate issue), although in the case of JSON, being treated as Unicode is relevant because of the escape codes that can be used in JSON string literals (although this does not make it necessary to validate UTF-8; it only makes it necessary to encode UTF-8 when an escape code is encountered).
Furthermore, when displaying text only for writing to a file, or to a terminal which is assumed to already have the correct character encoding (if you do not need to deal with alignment and stuff like that), you do not need to worry about UTF-8, and in fact is better that you don't; then it will use the same character encoding that it already is and will already be correct, whether it is UTF-8 or not (and you can avoid unnecessaily wasting time with validating UTF-8). (A program might require though that it is valid ASCII or extended ASCII (so e.g. UTF-16 will not work), but shouldn't need to care what the non-ASCII bytes mean.)
Unfortunately, some programming languages make it difficult.