4 ms·

> UTF-8 only encompasses 7-bit ASCII What? This is wrong. UTF-8 encodes a lot more than just ASCII. UTF-8 is compatible with ASCII in that all of the characte

by derleth 14y ago

> UTF-8 only encompasses 7-bit ASCII What? This is wrong. UTF-8 encodes a lot more than just ASCII. UTF-8 is compatible with ASCII in that all of the characters ASCII and Unicode have in common are represented the same way in ASCII and UTF-8. Going beyond ASCII involves the introduction of multi-byte representations in UTF-8, and that takes you smoothly (that is, no surrogate pairs) out into the entire rest of Unicode. As a bonus, it's always possible to verify that a given string of bytes is valid UTF-8, given that there is a nontrivial structure imposed on UTF-8 multi-byte encodings that is very unlikely to occur by chance in any non-UTF-8 sequence of bytes.

Firehed 14y ago
I think the point was that anything above 7-bit ASCII will be represented differently in Latin-1 vs UTF-8; i.e. ¢ (U+00A2) is rendered as 0xA2 in Latin1 and 0xC2A2 in UTF-8 - and 0xC2A2 in Latin1 will be displayed as Â¢. It gets far worse in 3-byte UTF8 characters, but I don't believe any of them exist natively in Latin1 (see: euro symbol) Assuming I'm reading these various character tables right, at least ;) So a more accurate version of what you quoted would be "UTF-8 and Latin-1 only overlap for 7-bit ASCII"
- kbolino 14y ago
  Not to detract from your points, all good, but: 0xC2A2 will be rendered as Â¢ only if it's encoded in UTF-16/UCS-2 big endian and misinterpreted as ISO-8859-1/Windows-1252. If it's encoded in little endian (much more common on Intel x86 computers), then it would be rendered as ¢Â when misinterpreted.
  - kelnos 14y ago
    That doesn't really make sense. If someone's intending to encode ¢ in utf-8, endianness does not come into play, as it's a stream of octets, not of anything larger that you can chunk such that you could swap bytes. At any rate, if you were to encode ¢ in UTF-16BE, it would be 0x00a2, not 0xc2a2. If a piece of software then misinterpreted it as latin1, likely you'd get nothing at all due to the embedded NUL. $ echo -n ¢ | iconv -f UTF-8 -t UTF-16BE | hexdump -C 00000000 00 a2 |..|
    - kbolino 14y ago
      "That doesn't really make sense." Indeed. I either completely misread the parent post, or else it said something different when I responded to it (knowing myself, I'm going with the former).

by derleth 14y ago

Firehed 14y ago

I think the point was that anything above 7-bit ASCII will be represented differently in Latin-1 vs UTF-8; i.e. ¢ (U+00A2) is rendered as 0xA2 in Latin1 and 0xC2A2 in UTF-8 - and 0xC2A2 in Latin1 will be displayed as Â¢. It gets far worse in 3-byte UTF8 characters, but I don't believe any of them exist natively in Latin1 (see: euro symbol) Assuming I'm reading these various character tables right, at least ;) So a more accurate version of what you quoted would be "UTF-8 and Latin-1 only overlap for 7-bit ASCII"

kbolino 14y ago
Not to detract from your points, all good, but: 0xC2A2 will be rendered as Â¢ only if it's encoded in UTF-16/UCS-2 big endian and misinterpreted as ISO-8859-1/Windows-1252. If it's encoded in little endian (much more common on Intel x86 computers), then it would be rendered as ¢Â when misinterpreted.
- kelnos 14y ago
  That doesn't really make sense. If someone's intending to encode ¢ in utf-8, endianness does not come into play, as it's a stream of octets, not of anything larger that you can chunk such that you could swap bytes. At any rate, if you were to encode ¢ in UTF-16BE, it would be 0x00a2, not 0xc2a2. If a piece of software then misinterpreted it as latin1, likely you'd get nothing at all due to the embedded NUL. $ echo -n ¢ | iconv -f UTF-8 -t UTF-16BE | hexdump -C 00000000 00 a2 |..|
  - kbolino 14y ago
    "That doesn't really make sense." Indeed. I either completely misread the parent post, or else it said something different when I responded to it (knowing myself, I'm going with the former).