5 ms·

UTF-8 bit by bit (2001)

76 points by networked 6y ago

klyrs 6y ago
This leaves a question: what of 0xFF? It can't be a "trailer", and has no trailing zero required of a "locomotive". No worries, it's simply forbidden.
- tialaramex 6y ago
  If you are processing UTF-8 and can't handle "forbidden" (for example you are in a language with no exception handling, or this is code which has a must-never-fail constraint) you should treat whatever forbidden sequence you've encountered as U+FFFD ("Replacement Character") and continue. The replacement character is commonly visually represented as a black diamond with a white question mark inside it, so humans will recognise that this means something went wrong, while a machine can't confuse U+FFFD with any letter, digit, white space, or other commonly used element it may be treating specially so this prevents many potential attacks on a parser. The article is unfortunate because it's actually handling a common non-UTF-8 encoding in which U+0000 is encoded differently so as to keep C-style null-terminated string semantics while allowing the NUL character. You should probably avoid the necessity for such semantics.
  - electrum 6y ago
    This special encoding of the null character is Modified UTF-8 and is used by various Java and JVM APIs and in Java class files. https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8
- rini17 6y ago
  The bytes FF FE or FE FF in the beginning of file are often used to indicate whole file is UTF-16 with "byte order mark", not UTF-8. I don't know of any other practical use, it's best to treat it as invalid.
puzzledobserver 6y ago
Here's a potentially random question: Is UTF-8 essentially a fancy way of representing arbitrary precision BigInts where the character-to-integer mapping is specified by Unicode, and which coincides with ASCII encodings on a certain range of numbers?
- 1ris 6y ago
  UTF-8 is not arbitrary precision. There is a artificial limit at 2^20. And Unicode does not map theses to characters. Unicode maps these to so called "code points". These "code points" then are mapped to charaters ("glypheme clusters" in unicode speak) in a very complicated way. A sequence of code points can be different kinds of normal forms, for example.
  - masklinn 6y ago
    > UTF-8 is not arbitrary precision. There is a artificial limit at 2^20. Since that limit is artificial it can safely be ignored. UTF-8 was designed as a 32b encoding. I don't think it is arbitrary-precision either though, since it was also designed to be self-synchronising. > glypheme Grapheme. A grapheme is more or less a user-perceived character, a "grapheme cluster" is a group (cluster) of codepoints roughly corresponding to a grapheme. > A sequence of code points can be different kinds of normal forms, for example. That has limited relevance, since not all clusters exist in precomposed forms they have to be handled regardless of normalisation, there's just some redundancies.
- masklinn 6y ago
  UTF-8 is a "fancy" way of representing 32 bit integers in a self-synchronising byte format. I can see an extension to 7 bytes (by making the leading byte 11111110 which I think would still be unambiguous) going up to 37 bits, but at 8 bytes you'd start getting collisions in the byte patterns and would lose the self-synchronising properties.
  - kstenerud 6y ago
    You could in theory extend the pattern up to 42 bits: 0xxxxxxx 110xxxxx 10xxxxxx 1110xxxx 10xxxxxx 10xxxxxx 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
stormdennis 6y ago
I found this analogy of trains and railcars good but I'm lost on this bit (110)00000 (10)000000 => C0 80 EDIT: OK I asssume he's referring to C and octal here and not a hexadecimal and decimal numbers?
- duskwuff 6y ago
  Binary 11000000 10000000 = hexadecimal C0 80.
  - stormdennis 6y ago
    Thank you, got it now! I think he might have been clearer if he'd said "To represent a NUL byte without any physical NUL bytes, we don't discard the indicators and instead treat it like a character above ASCII, which must be a minimum two bytes long 11000000 10000000 => hexadecimal C0 80."
dependenttypes 6y ago
One small fun fact: utf8 uses big-endian.

76 points by networked 6y ago

This leaves a question: what of 0xFF? It can't be a "trailer", and has no trailing zero required of a "locomotive". No worries, it's simply forbidden.

tialaramex 6y ago
If you are processing UTF-8 and can't handle "forbidden" (for example you are in a language with no exception handling, or this is code which has a must-never-fail constraint) you should treat whatever forbidden sequence you've encountered as U+FFFD ("Replacement Character") and continue. The replacement character is commonly visually represented as a black diamond with a white question mark inside it, so humans will recognise that this means something went wrong, while a machine can't confuse U+FFFD with any letter, digit, white space, or other commonly used element it may be treating specially so this prevents many potential attacks on a parser. The article is unfortunate because it's actually handling a common non-UTF-8 encoding in which U+0000 is encoded differently so as to keep C-style null-terminated string semantics while allowing the NUL character. You should probably avoid the necessity for such semantics.
- electrum 6y ago
  This special encoding of the null character is Modified UTF-8 and is used by various Java and JVM APIs and in Java class files. https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8
rini17 6y ago
The bytes FF FE or FE FF in the beginning of file are often used to indicate whole file is UTF-16 with "byte order mark", not UTF-8. I don't know of any other practical use, it's best to treat it as invalid.

puzzledobserver 6y ago

Here's a potentially random question: Is UTF-8 essentially a fancy way of representing arbitrary precision BigInts where the character-to-integer mapping is specified by Unicode, and which coincides with ASCII encodings on a certain range of numbers?

1ris 6y ago
UTF-8 is not arbitrary precision. There is a artificial limit at 2^20. And Unicode does not map theses to characters. Unicode maps these to so called "code points". These "code points" then are mapped to charaters ("glypheme clusters" in unicode speak) in a very complicated way. A sequence of code points can be different kinds of normal forms, for example.
- masklinn 6y ago
  > UTF-8 is not arbitrary precision. There is a artificial limit at 2^20. Since that limit is artificial it can safely be ignored. UTF-8 was designed as a 32b encoding. I don't think it is arbitrary-precision either though, since it was also designed to be self-synchronising. > glypheme Grapheme. A grapheme is more or less a user-perceived character, a "grapheme cluster" is a group (cluster) of codepoints roughly corresponding to a grapheme. > A sequence of code points can be different kinds of normal forms, for example. That has limited relevance, since not all clusters exist in precomposed forms they have to be handled regardless of normalisation, there's just some redundancies.
masklinn 6y ago
UTF-8 is a "fancy" way of representing 32 bit integers in a self-synchronising byte format. I can see an extension to 7 bytes (by making the leading byte 11111110 which I think would still be unambiguous) going up to 37 bits, but at 8 bytes you'd start getting collisions in the byte patterns and would lose the self-synchronising properties.
- kstenerud 6y ago
  You could in theory extend the pattern up to 42 bits: 0xxxxxxx 110xxxxx 10xxxxxx 1110xxxx 10xxxxxx 10xxxxxx 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

stormdennis 6y ago

I found this analogy of trains and railcars good but I'm lost on this bit (110)00000 (10)000000 => C0 80 EDIT: OK I asssume he's referring to C and octal here and not a hexadecimal and decimal numbers?

duskwuff 6y ago
Binary 11000000 10000000 = hexadecimal C0 80.
- stormdennis 6y ago
  Thank you, got it now! I think he might have been clearer if he'd said "To represent a NUL byte without any physical NUL bytes, we don't discard the indicators and instead treat it like a character above ASCII, which must be a minimum two bytes long 11000000 10000000 => hexadecimal C0 80."

dependenttypes 6y ago

One small fun fact: utf8 uses big-endian.