5 ms·
So Twitter allows 140 (UTF-8?) characters, regardless of the number bytes? The article wasn't clear about this.
by cpeterso 15y ago
So Twitter allows 140 (UTF-8?) characters, regardless of the number bytes? The article wasn't clear about this.
- waffle_ss 15y agoHere's the Twitter dev article that I used for designing my URL shortener: https://dev.twitter.com/docs/counting-characters#Twitter_Character_Encoding https://dev.twitter.com/docs/counting-characters#Twitter_Cha... Basically, they use the Normalization Form C of Unicode normalization which counts code points, not UTF-8 bytes.
- wisty 15y agoA slight modification to your tldr (it's a little confusing to me as I'm not so familiar with unicode): Basically, they use count code points, not UTF-8 bytes. Before they count the code points, they normalize using the Normalization Form C of Unicode (NFC), which aims to combine diacritics, so the "é" in "café" counts as 1 code point. If they used NFD to normalize, "é" would be normalized into 2 code points - "e" and a diacritic mark. If they didn't normalize, then it would be client dependent. Normalization is distinct from encoding. A unicode string can be normalized in several different ways (including NFC and NFD), which changes the actual unicode code-points). Each normalized form can be encoded using several different encodings (i.e. utf-8, utf-32, latin1, etc). Normalization affects both the number of code points, and subsequently the number of bytes. Encoding (unicode -> bytes) changes the number of bytes, but should not affect the number of code points - code points should not be influenced by the encoding. Recapping - Twitter does not count bytes. They count code points, after the code points have been normalized (combining diacritics with characters), so message length should not depend on the client. Hmm, that wasn't so succinct.
- masklinn 15y agoTwitters allows 140 codepoints of a sequence in normalization form C.
- adgar 15y agoWait, really? I thought normalization form C was the form where composite characters were always used when possible. Why restrict by the number of codepoints (vs characters) if you're explicitly going to use the form which goes out of its way to use multi-codepoint characters? The only reason I can think of is that they internally use UTF-32, so counting codepoints is more efficient. But I thought they used UTF-8. Edit: the other reason I can think of is that conversion to normalization form C already counts the codepoints. Though I can't imagine making it also count characters would be nontrivial.
- wisty 15y agoMost likely, because Twitter is a microblog, and 140 character posts is it's thing. I'm sure it had some historical technical limitation to 140 ASCII or maybe 70 utf-8 chars (or something else logical), but they probably had to accomodate people who wanted to use non-English characters in a post and not get a lecture on unicode encoding; and some slightly offensive "so ... people like you only get 70 chars" message.
- adgar 15y agoYou misunderstand. I'm wondering why they allow only 140 codepoints and not characters. Even in Unicode, 1 codepoint != 1 character.
- wisty 15y agoSorry, my Unicode is a bit weak. I think Twitter should use whichever usually gives the user the most characters, to prevent them from getting burnt. I think in many cases, the normalized form is more permissive, as it puts "character plus diacritic" together into one character. In a language with lots of diacritics, the number of codepoints might be more than the number of normalized characters (depending on the client). You wouldn't want to allow (say) ~70 Korean characters on one OS, and 140 on another, just because they use different codepoints to represent certain characters - one with character then diacritic (2 codepoints?), another with both crammed together in one codepoint. But as I said, I'm not a unicode guru (and I don't know much about Korean, I just saw it as an example). This might be wrong.