12 ms·

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 s

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

deleted 14y ago
[deleted]
- pjscott 14y ago
  If you make an API, please don't do this. If you have to include indexes into a Unicode string, then make them indexes into a binary string in a known encoding. This works equally well for all encodings, and won't tempt people to do something as preposterous as using UCS-2.
- mikeash 14y ago
  Such an API could easily indicate bytes in a UTF-8 string instead.
- marshray 14y ago
  You will never define an unambiguous way to index into a Unicode string by "characters" with less than 10 pages of specification. When two sides of the API interpret this contract differently, a security vulnerability is a very possible outcome. Use unambiguous delimiters, JSON, XML, or even byte offsets into normalized UTF-8 instead. But don't do this please.
SeoxyS 14y ago
It's not a common use case, but I've had to do it. Luckily, it's fairly easy: int count_multibytes_encountered(char *text, unsigned long len) { int count = 0; for (int i = 0; i < len; i++) { if ((*(text+i) & 0b10000000) == 0b10000000 && // check if it's a multi-char byte (~*(text+i) & 0b01000000) == 0b01000000) { // and check that it's not a leading byte count++; } } return count; }
pjscott 14y ago
Be careful not to confuse UTF-16 and UCS-2. UTF-16 is a variable-width encoding, so using it doesn't actually make counting anything easier. UCS-2 is fixed-width, and evil. With UCS-2 you can easily count the number of code points, as long as they fall in the BMP (!), but this is not the same as counting graphemes.
est 14y ago
> Why do you want to count Unicode characters? Because, there are other countries which use more than English language? I fucking hate you ascii-centric ignorant morons sometimes, you know, for example - display welcome message character by character fro left to right - Extract the first character because it's always the surname - catch two non-ascii keyword and find its index in a string In the first example, should I just put byte by byte, which displays as garbage, and suddenly three bytes become a recognizable character?
- dietrichepp 14y ago
  I think you misunderstand. I wasn't asking why Unicode characters should be counted instead of bytes or ASCII characters, I was asking why you would even want to count characters at all. > I fucking hate you ascii-centric ignorant morons Nice. > You ignorant, arrogant fuck. This is why I quit posting under an alias, so I wouldn't be tempted to say such things. > display welcome message character by character fro left to right UTF-16/UCS-4/UCS-2 doesn't solve anything here. Counting characters doesn't help. For example, imagine if you try to print Korean character-by-character. You might get some garbage like this: ᄋ 아 안 안ᄂ 안녀 안녕 안녕ᄒ 안녕하 안녕하ᄉ 안녕하세 안녕하세ᄋ 안녕하세요 Fixed width encodings do not solve this problem, and UTF-8 does not make this problem more difficult. I am honestly curious why you would need to count characters -- at all -- except for posting to Twitter. Splitting on characters is garbage. (This example was done in Python 3, so everything is properly encoded, and there is no need to use the 'u' prefix. The 'u' prefix is a nop in Python 3. It is only there for Python 2.x compatibility.) >>> x '안녕하세요' >>> x[2:4] 'ᆫᄂ' I tried in the Google Chrome console, too: > '안녕하세요'.substr(2,2) "하세" > '안녕하세요'.substr(2,2) "ᆫᄂ" I'm not even leaving the BMP and it's broken! You seem to be blaming encoding issues but I don't have any issues with encoding. It doesn't matter if Chrome uses UCS-2 or Python uses UCS-4 or UCS-2, what's happening here is entirely expected, and it has everything to do with Jamo and nothing to do with encodings. >>> a = '안녕하세요' >>> b = '안녕하세요' # They only look the same >>> len(a) 5 >>> len(b) 12 >>> def p(x): ... return ' '.join( 'U+{:04X}'.format(ord(c)) for c in x) >>> print(' '.join('U+{:04X}'.format(ord(c)) for c in b)) >>> print(p(a)) U+C548 U+B155 U+D558 U+C138 U+C694 >>> print(p(b)) U+110B U+1161 U+11AB U+1102 U+1167 U+11BC U+1112 U+1161 U+1109 U+1166 U+110B U+116D See? Expected, broken behavior you get when splitting on character boundaries. If you think you can split on character boundaries, you are living in an ASCII world. Unicode does not work that way. Don't think that normalization will solve anything either. (Okay, normalization solves some problems. But it is not a panacea. Some languages have grapheme clusters that cannot be precomposed.) Fixed-width may be faster for splitting on character boundaries, but splitting on character boundaries only works in the ASCII world.
Daniel_Newby 14y ago
> Why do you want to count Unicode characters? Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information. > Why do you care if it is fast to do so? Efficient full text search that can ignore decorative combining characters.
- meatmanek 14y ago
  > Text editing and rendering Unless you're working entirely in fixed point characters (and you probably aren't, given that even fixed-width systems like terminal emulators use double-wide glyphs sometimes), you need to know the value of each character to know its width. That involves the same linear scan over the string that is required to calculate the number of glyphs in a variable-width encoding.
- pjscott 14y ago
  How does fast character counting help with full-text search?
  - Daniel_Newby 14y ago
    The best search algorithms can skip ahead upon a mismatch. A variable-length encoding requires branch instructions in the inner loop, leading to pipeline flushes and potentially dramatic slow down.
    - dietrichepp 14y ago
      This is incorrect. Searching text with a variable-length encoding does not require extra branch instructions. If you're searching through UTF-8 text, you can just pretend it's a bunch of bytes and search through that. This isn't counting problems with normalization, of course. You will have to put your needle and haystack both into the same normalization form before searching. But you had to do that anyway.
- andolanra 14y ago
  If you implement naïve Aho-Corasick text search over one-byte characters, it works without modification on UTF-8 text. It does not ignore combining characters, but UCS-2 also features combining characters (c.f. other comments in this same thread), so no matter what encoding you use, you must first normalize the Unicode text and the search string before you compare for equivalence (or compatibility, which is a looser notion than equality for Unicode code point sequences.)

deleted 14y ago

[deleted]

pjscott 14y ago
If you make an API, please don't do this. If you have to include indexes into a Unicode string, then make them indexes into a binary string in a known encoding. This works equally well for all encodings, and won't tempt people to do something as preposterous as using UCS-2.
mikeash 14y ago
Such an API could easily indicate bytes in a UTF-8 string instead.
marshray 14y ago
You will never define an unambiguous way to index into a Unicode string by "characters" with less than 10 pages of specification. When two sides of the API interpret this contract differently, a security vulnerability is a very possible outcome. Use unambiguous delimiters, JSON, XML, or even byte offsets into normalized UTF-8 instead. But don't do this please.

SeoxyS 14y ago

It's not a common use case, but I've had to do it. Luckily, it's fairly easy: int count_multibytes_encountered(char *text, unsigned long len) { int count = 0; for (int i = 0; i < len; i++) { if ((*(text+i) & 0b10000000) == 0b10000000 && // check if it's a multi-char byte (~*(text+i) & 0b01000000) == 0b01000000) { // and check that it's not a leading byte count++; } } return count; }

pjscott 14y ago

Be careful not to confuse UTF-16 and UCS-2. UTF-16 is a variable-width encoding, so using it doesn't actually make counting anything easier. UCS-2 is fixed-width, and evil. With UCS-2 you can easily count the number of code points, as long as they fall in the BMP (!), but this is not the same as counting graphemes.

est 14y ago

> Why do you want to count Unicode characters? Because, there are other countries which use more than English language? I fucking hate you ascii-centric ignorant morons sometimes, you know, for example - display welcome message character by character fro left to right - Extract the first character because it's always the surname - catch two non-ascii keyword and find its index in a string In the first example, should I just put byte by byte, which displays as garbage, and suddenly three bytes become a recognizable character?

dietrichepp 14y ago
I think you misunderstand. I wasn't asking why Unicode characters should be counted instead of bytes or ASCII characters, I was asking why you would even want to count characters at all. > I fucking hate you ascii-centric ignorant morons Nice. > You ignorant, arrogant fuck. This is why I quit posting under an alias, so I wouldn't be tempted to say such things. > display welcome message character by character fro left to right UTF-16/UCS-4/UCS-2 doesn't solve anything here. Counting characters doesn't help. For example, imagine if you try to print Korean character-by-character. You might get some garbage like this: ᄋ 아 안 안ᄂ 안녀 안녕 안녕ᄒ 안녕하 안녕하ᄉ 안녕하세 안녕하세ᄋ 안녕하세요 Fixed width encodings do not solve this problem, and UTF-8 does not make this problem more difficult. I am honestly curious why you would need to count characters -- at all -- except for posting to Twitter. Splitting on characters is garbage. (This example was done in Python 3, so everything is properly encoded, and there is no need to use the 'u' prefix. The 'u' prefix is a nop in Python 3. It is only there for Python 2.x compatibility.) >>> x '안녕하세요' >>> x[2:4] 'ᆫᄂ' I tried in the Google Chrome console, too: > '안녕하세요'.substr(2,2) "하세" > '안녕하세요'.substr(2,2) "ᆫᄂ" I'm not even leaving the BMP and it's broken! You seem to be blaming encoding issues but I don't have any issues with encoding. It doesn't matter if Chrome uses UCS-2 or Python uses UCS-4 or UCS-2, what's happening here is entirely expected, and it has everything to do with Jamo and nothing to do with encodings. >>> a = '안녕하세요' >>> b = '안녕하세요' # They only look the same >>> len(a) 5 >>> len(b) 12 >>> def p(x): ... return ' '.join( 'U+{:04X}'.format(ord(c)) for c in x) >>> print(' '.join('U+{:04X}'.format(ord(c)) for c in b)) >>> print(p(a)) U+C548 U+B155 U+D558 U+C138 U+C694 >>> print(p(b)) U+110B U+1161 U+11AB U+1102 U+1167 U+11BC U+1112 U+1161 U+1109 U+1166 U+110B U+116D See? Expected, broken behavior you get when splitting on character boundaries. If you think you can split on character boundaries, you are living in an ASCII world. Unicode does not work that way. Don't think that normalization will solve anything either. (Okay, normalization solves some problems. But it is not a panacea. Some languages have grapheme clusters that cannot be precomposed.) Fixed-width may be faster for splitting on character boundaries, but splitting on character boundaries only works in the ASCII world.

Daniel_Newby 14y ago

> Why do you want to count Unicode characters? Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information. > Why do you care if it is fast to do so? Efficient full text search that can ignore decorative combining characters.

meatmanek 14y ago
> Text editing and rendering Unless you're working entirely in fixed point characters (and you probably aren't, given that even fixed-width systems like terminal emulators use double-wide glyphs sometimes), you need to know the value of each character to know its width. That involves the same linear scan over the string that is required to calculate the number of glyphs in a variable-width encoding.
pjscott 14y ago
How does fast character counting help with full-text search?
- Daniel_Newby 14y ago
  The best search algorithms can skip ahead upon a mismatch. A variable-length encoding requires branch instructions in the inner loop, leading to pipeline flushes and potentially dramatic slow down.
  - dietrichepp 14y ago
    This is incorrect. Searching text with a variable-length encoding does not require extra branch instructions. If you're searching through UTF-8 text, you can just pretend it's a bunch of bytes and search through that. This isn't counting problems with normalization, of course. You will have to put your needle and haystack both into the same normalization form before searching. But you had to do that anyway.
andolanra 14y ago
If you implement naïve Aho-Corasick text search over one-byte characters, it works without modification on UTF-8 text. It does not ignore combining characters, but UCS-2 also features combining characters (c.f. other comments in this same thread), so no matter what encoding you use, you must first normalize the Unicode text and the search string before you compare for equivalence (or compatibility, which is a looser notion than equality for Unicode code point sequences.)