6 ms·

Chinese Twitter users live in a density 2x to 8x their English counterparts.

35 points by audreyt 17y ago

lsb 17y ago
Michael Mitzenmacher, at Harvard, had a paper in the 2003 IEEE Data Compression Conference that gave empirical evidence that translations compressed to roughly similar sizes (using the Bible and the EU texts), but had wildly varying sizes uncompressed; this correlates well with linguistic theories. ftp://ftp.deas.harvard.edu/techreports/tr-12-02.ps.gz
- est 17y ago
  similar post on reddit: http://www.reddit.com/r/programming/comments/9sjt0/rudebox_by_alcatraz_4091_bytes_of_magic/c0e91bl?context=1 http://www.reddit.com/r/programming/comments/9sjt0/rudebox_b...
jrockway 17y ago
If you are willing to do a bit of encoding/decoding, then you can map 2 or 3 latin-1 characters onto one unicode codepoint, and then tweet with that instead. My understanding of UTF-8 indicates that you can actually represent any number as one character, but somewhere in the xterm / firefox / twitter pipeline, that gets fucked up. I think I have some code on github for this, actually: http://gist.github.com/191446 http://gist.github.com/191446 The idea is to pack any utf-8 string into one character. It works for about 3 or 4 ASCII characters, but I think this is a perl bug rather than some fundamental limitation. Patches welcome. (As an aside, I am always pleased when I get to use the (>>=) operator in Perl. And yes, I do pronounce it "bind" and not "right-shift-equals" ;)
- tlrobinson 17y ago
  UTF-8 encodes any Unicode character using between 1 and 4 bytes. Also, some byte sequences aren't valid UTF-8. I don't think it's a Perl bug.
megaduck 17y ago
This is one of the reasons that texting is so darn popular in China, and email is not. Texting is fast, and you can pack so much info into a single SMS that you almost never need to send anything longer. The same goes for books and essays. Chinese books and magazines are often shorter, just because the information density is so high. It's a neat feature of the language. However, sometimes it can be demoralizing when you spend all evening writing something and realize that you've only produced a single page of text.
- jbert 17y ago
  > This is one of the reasons that texting is so darn popular in China, and email is not. Texting is fast I don't see why texting in Chinese should be faster than in English. Don't you need to push a similar number of bits through the numeric keypad? I'd imagine that to be the limiting factor. i.e. English texters need to send more chars, but they're choosing from fewer chars so need fewer keypresses to select each one. Chinese texters need to send fewer chars, but I imagine they need to make more keypresses to select each char.
  - garply 17y ago
    Yes, you need 1-4 keypresses per char, but the language itself is significantly more compressed than English. The equivalent of an English word is 1 to 2 chars (with more frequent words usually being one character - just as frequent English words tend to be shorter), but the number of "words" you use to express the same idea is generally fewer, primarily because the language is a lot more ambiguous than English (e.g., there is no 'the', 'a', or 'an', and subjects are often dropped - it is left to the speakers to fill in the blanks from context).
    - jbert 17y ago
      OK, you need 1-3 keypresses for English. Ah, of course, it's tree navigation. The number of keypresses required to enter the character is O(log(number-of-chars-in-alphabet)). So having lots of characters is a win for input speed. Thanks.
  - 17y ago
deleted 17y ago
[deleted]
johannchiang 17y ago
Kaifu Lee mentioned the same observation in a talk. Chinese news titles usually bear sufficient information that Google news in Chinese doesn't need excerpt as counterpart in English. A Tweet in Chinese could be an essay.

35 points by audreyt 17y ago

lsb 17y ago

Michael Mitzenmacher, at Harvard, had a paper in the 2003 IEEE Data Compression Conference that gave empirical evidence that translations compressed to roughly similar sizes (using the Bible and the EU texts), but had wildly varying sizes uncompressed; this correlates well with linguistic theories. ftp://ftp.deas.harvard.edu/techreports/tr-12-02.ps.gz

est 17y ago
similar post on reddit: http://www.reddit.com/r/programming/comments/9sjt0/rudebox_by_alcatraz_4091_bytes_of_magic/c0e91bl?context=1 http://www.reddit.com/r/programming/comments/9sjt0/rudebox_b...

jrockway 17y ago

If you are willing to do a bit of encoding/decoding, then you can map 2 or 3 latin-1 characters onto one unicode codepoint, and then tweet with that instead. My understanding of UTF-8 indicates that you can actually represent any number as one character, but somewhere in the xterm / firefox / twitter pipeline, that gets fucked up. I think I have some code on github for this, actually: http://gist.github.com/191446 http://gist.github.com/191446 The idea is to pack any utf-8 string into one character. It works for about 3 or 4 ASCII characters, but I think this is a perl bug rather than some fundamental limitation. Patches welcome. (As an aside, I am always pleased when I get to use the (>>=) operator in Perl. And yes, I do pronounce it "bind" and not "right-shift-equals" ;)

tlrobinson 17y ago
UTF-8 encodes any Unicode character using between 1 and 4 bytes. Also, some byte sequences aren't valid UTF-8. I don't think it's a Perl bug.

megaduck 17y ago

This is one of the reasons that texting is so darn popular in China, and email is not. Texting is fast, and you can pack so much info into a single SMS that you almost never need to send anything longer. The same goes for books and essays. Chinese books and magazines are often shorter, just because the information density is so high. It's a neat feature of the language. However, sometimes it can be demoralizing when you spend all evening writing something and realize that you've only produced a single page of text.

jbert 17y ago
> This is one of the reasons that texting is so darn popular in China, and email is not. Texting is fast I don't see why texting in Chinese should be faster than in English. Don't you need to push a similar number of bits through the numeric keypad? I'd imagine that to be the limiting factor. i.e. English texters need to send more chars, but they're choosing from fewer chars so need fewer keypresses to select each one. Chinese texters need to send fewer chars, but I imagine they need to make more keypresses to select each char.
- garply 17y ago
  Yes, you need 1-4 keypresses per char, but the language itself is significantly more compressed than English. The equivalent of an English word is 1 to 2 chars (with more frequent words usually being one character - just as frequent English words tend to be shorter), but the number of "words" you use to express the same idea is generally fewer, primarily because the language is a lot more ambiguous than English (e.g., there is no 'the', 'a', or 'an', and subjects are often dropped - it is left to the speakers to fill in the blanks from context).
  - jbert 17y ago
    OK, you need 1-3 keypresses for English. Ah, of course, it's tree navigation. The number of keypresses required to enter the character is O(log(number-of-chars-in-alphabet)). So having lots of characters is a win for input speed. Thanks.
- 17y ago

deleted 17y ago

[deleted]

johannchiang 17y ago

Kaifu Lee mentioned the same observation in a talk. Chinese news titles usually bear sufficient information that Google news in Chinese doesn't need excerpt as counterpart in English. A Tweet in Chinese could be an essay.