4 ms·

UTF-8 characters can be composed of at most 4 bytes, plus a NUL terminator makes 5. I have no idea if this is true but it’s plausible enough for a Unicode parse

UTF-8 characters can be composed of at most 4 bytes, plus a NUL terminator makes 5. I have no idea if this is true but it’s plausible enough for a Unicode parser!

rurban 6y ago
On the character level yes, but this is for strings, so looks totally broken to me.
- rusk 6y ago
  The len is being passed to malloc which I recall takes number of bytes, so this explanation seems plausible. I very much doubt memory allocation is ”innacurate”, as per the comment ...
SAI_Peregrinus 6y ago
Not UTF-8 characters, UTF-8 code points. UTF-8 characters (more properly "extended grapheme clusters") can be of any length. " " is one character, 25 bytes long (0xf0,0x9f,0x91,0xa8,0xe2,0x80,0x8d,0xf0,0x9f,0x91,0xa8,0xe2,0x80,0x8d,0xf0,0x9f,0x91,0xa7,0xe2,0x80,0x8d,0xf0,0x9f,0x91,0xa7) (it won't show correctly on HN, since HN strips some Unicode). If you assume that code point = character, your parser is buggy and doesn't comply with unicode specifications.
- floatingatoll 6y ago
  I don’t presume to know enough about Unicode to code parsers it, but I’m happy to accept the correction!
- zepearl 6y ago
  Uuuhh, this is an extremely important remark/correction for me, thank you. So far I was thinking as well that any "character" could be saved ("encoded"?) as UTF-8 by using max 4 bytes... .
  - floatingatoll 6y ago
    Any UTF-8 code point can be stored in 4 bytes or less, and so as long as you don’t care how many characters your string ends up displaying to the user or try to dissect the string by “characters” somehow, you’ll probably be fine; it’s just a series of valid UTF-8 code points that have meaning you don’t care about at your storage layer. If you’re tokenizing UTF-8 strings or trying to count “displayed” characters, then you also have to watch out for zalgo text that can use unlimited chains of code points and combiners to construct a single phrase out of potentially tens or hundreds of code points. Think it as a recursion limit of sorts. I feel that Unicode and cryptography share the aspect of “don’t try to write your own code to deal with it, because you’re certain to miss something important”.
    - zepearl 6y ago
      Thanks! I'll have to review my code... :P
- anonymfus 6y ago
  > Not UTF-8 characters, UTF-8 code points. UTF-8 characters (more properly "extended grapheme clusters") can be of any length. Technically the only thing UTF-8 encodes is Unicode code points, as higher level abstractions like grapheme clusters are irrelevant for it. UTF-8 parser's job is to turn a sequence of octets to a sequence of code points. So-called "user-perceived characters" approximated as grapheme clusters is a relatively recent innovation in Unicode terminology. Compare the modern version of the spec (https://www.unicode.org/reports/tr29/ https://www.unicode.org/reports/tr29/) since Unicode 5.1.0 (2008): > It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a user-perceived character: users think of it as a single character, yet is actually represented by two Unicode code points. These user-perceived characters are approximated by what is called a grapheme cluster, which can be determined programmatically. With the previous one (https://www.unicode.org/reports/tr29/tr29-11.html https://www.unicode.org/reports/tr29/tr29-11.html) from Unicode 5.0.0 (2006): > One or more Unicode characters may make up what the user thinks of as a character or basic unit of the language. To avoid ambiguity with the computer use of the term character, this is called a grapheme cluster. For example, “G” + acute-accent is a grapheme cluster: it is thought of as a single character by users, yet is actually represented by two Unicode code points. So you can notice "the computer use of the term character" in both. Notice also that these grave and acute accents used as examples existed in pre-Unicode encodings too, and we still call them "combining characters". Hence if we adhere to that old "computer use of the term character" then "UTF-8 characters" are Unicode code points, and "Unicode characters" don't exist; but if we use a modern colloquial meaning of characters as Unicode's "user-perceived characters" then "UTF-8 characters" don't exist.
- Izkata 6y ago
  For those of us not fully in on the technical details, is your example basically invisible Zalgo Text, or is this something more subtle?
  - deleted 6y ago
    [deleted]
  - SAI_Peregrinus 6y ago
    It's a single emoji: "Family: Man, Man, Girl, Girl". Sequence is "Man, Zero-Width Joiner (ZWJ), Man, ZWJ, Girl, ZWJ, Girl". Picked because I looked up "Man" and "Girl" first, and just repeating the two is easy. https://emojipedia.org/family-man-man-girl-girl/ https://emojipedia.org/family-man-man-girl-girl/

UTF-8 characters can be composed of at most 4 bytes, plus a NUL terminator makes 5. I have no idea if this is true but it’s plausible enough for a Unicode parser!

rurban 6y ago

On the character level yes, but this is for strings, so looks totally broken to me.

rusk 6y ago
The len is being passed to malloc which I recall takes number of bytes, so this explanation seems plausible. I very much doubt memory allocation is ”innacurate”, as per the comment ...

SAI_Peregrinus 6y ago

Not UTF-8 characters, UTF-8 code points. UTF-8 characters (more properly "extended grapheme clusters") can be of any length. " " is one character, 25 bytes long (0xf0,0x9f,0x91,0xa8,0xe2,0x80,0x8d,0xf0,0x9f,0x91,0xa8,0xe2,0x80,0x8d,0xf0,0x9f,0x91,0xa7,0xe2,0x80,0x8d,0xf0,0x9f,0x91,0xa7) (it won't show correctly on HN, since HN strips some Unicode). If you assume that code point = character, your parser is buggy and doesn't comply with unicode specifications.

floatingatoll 6y ago
I don’t presume to know enough about Unicode to code parsers it, but I’m happy to accept the correction!
zepearl 6y ago
Uuuhh, this is an extremely important remark/correction for me, thank you. So far I was thinking as well that any "character" could be saved ("encoded"?) as UTF-8 by using max 4 bytes... .
- floatingatoll 6y ago
  Any UTF-8 code point can be stored in 4 bytes or less, and so as long as you don’t care how many characters your string ends up displaying to the user or try to dissect the string by “characters” somehow, you’ll probably be fine; it’s just a series of valid UTF-8 code points that have meaning you don’t care about at your storage layer. If you’re tokenizing UTF-8 strings or trying to count “displayed” characters, then you also have to watch out for zalgo text that can use unlimited chains of code points and combiners to construct a single phrase out of potentially tens or hundreds of code points. Think it as a recursion limit of sorts. I feel that Unicode and cryptography share the aspect of “don’t try to write your own code to deal with it, because you’re certain to miss something important”.
  - zepearl 6y ago
    Thanks! I'll have to review my code... :P
anonymfus 6y ago
> Not UTF-8 characters, UTF-8 code points. UTF-8 characters (more properly "extended grapheme clusters") can be of any length. Technically the only thing UTF-8 encodes is Unicode code points, as higher level abstractions like grapheme clusters are irrelevant for it. UTF-8 parser's job is to turn a sequence of octets to a sequence of code points. So-called "user-perceived characters" approximated as grapheme clusters is a relatively recent innovation in Unicode terminology. Compare the modern version of the spec (https://www.unicode.org/reports/tr29/ https://www.unicode.org/reports/tr29/) since Unicode 5.1.0 (2008): > It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a user-perceived character: users think of it as a single character, yet is actually represented by two Unicode code points. These user-perceived characters are approximated by what is called a grapheme cluster, which can be determined programmatically. With the previous one (https://www.unicode.org/reports/tr29/tr29-11.html https://www.unicode.org/reports/tr29/tr29-11.html) from Unicode 5.0.0 (2006): > One or more Unicode characters may make up what the user thinks of as a character or basic unit of the language. To avoid ambiguity with the computer use of the term character, this is called a grapheme cluster. For example, “G” + acute-accent is a grapheme cluster: it is thought of as a single character by users, yet is actually represented by two Unicode code points. So you can notice "the computer use of the term character" in both. Notice also that these grave and acute accents used as examples existed in pre-Unicode encodings too, and we still call them "combining characters". Hence if we adhere to that old "computer use of the term character" then "UTF-8 characters" are Unicode code points, and "Unicode characters" don't exist; but if we use a modern colloquial meaning of characters as Unicode's "user-perceived characters" then "UTF-8 characters" don't exist.
Izkata 6y ago
For those of us not fully in on the technical details, is your example basically invisible Zalgo Text, or is this something more subtle?
- deleted 6y ago
  [deleted]
- SAI_Peregrinus 6y ago
  It's a single emoji: "Family: Man, Man, Girl, Girl". Sequence is "Man, Zero-Width Joiner (ZWJ), Man, ZWJ, Girl, ZWJ, Girl". Picked because I looked up "Man" and "Girl" first, and just repeating the two is easy. https://emojipedia.org/family-man-man-girl-girl/ https://emojipedia.org/family-man-man-girl-girl/