4 ms·

Fun story about emoji and unicode: I have a writing app I've developed for iOS, and a while ago received a bug report from someone saying the app crashed every

Fun story about emoji and unicode: I have a writing app I've developed for iOS, and a while ago received a bug report from someone saying the app crashed every time they tried to open their document. I couldn't figure out what was going on but after eventually receiving a copy of the document, I discovered that the crash was due to my incorrect handling of emoji characters. Most string classes, including NSString in iOS, typically represent strings as sequences of 16-bit characters. With 65,536 possible combinations, this is perfectly adequate to support the characters from all major languages in use worldwide [1], plus emoji. Unicode does however allow for values outside of this range, which are represented by "surrogate" pairs of 16-bit values in the range D800-DFFF. It was these I was handling incorrectly, and had missed in my testing. But if there's space in the 0-65,535 range to represent emoji, why would it be stored outside of this? Well, it turns out that the initial implementation of emoji in iOS used the "private use area" of this range to encode the character values [2]. NTT DoCoMo also had their own (incompatible) way of encoding emoji in the private use area [3]. When attempts were made to formally standardise on emoji in unicode, they couldn't use the private use area, and ended up going with values above 65,535. If unicode had stuck to representing existing characters and symbols and said no to requests for stuff like emoji and klingon, string representation in modern software could have been kept a whole lot simpler. Joel Spolsky has an excellent explanation of unicode and encoding issues here: http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html [1] To the best of my knowledge; correct me if I'm wrong [2] http://openradar.appspot.com/6402446 http://openradar.appspot.com/6402446 [3] http://web.archive.org/web/20080216230900/http://www.nttdocomo.co.jp/english/service/imode/make/content/pictograph/basic/index.html http://web.archive.org/web/20080216230900/http://www.nttdoco...

btn 13y ago
this is perfectly adequate to support the characters from all major languages in use worldwide, plus emoji Emoji characters are in the range U+1F300..U+1F5FF, which is outside the BMP (up to U+FFFF), and will be encoded with surrogate pairs in UTF-16. Also, even within the BMP, assuming one 16-bit UTF-16 block is the same as one "character" is dangerous, as combining characters (for example) should arguably not be separated from their base character. If unicode had stuck to representing existing characters and symbols and said no to requests for stuff like emoji and klingon, string representation in modern software could have been kept a whole lot simpler. Emoji and Klingon are trivial to deal with in comparison to the issues that can appear with "real" languages when you start making assumptions (cf. combining characters, normalisation forms, CJK unification and compatibility blocks, presentation forms, collation, etc.)
vorg 13y ago
> With 65,536 possible combinations, this is perfectly adequate to support the characters from all major languages in use worldwide [1] It depends on how you represent characters. Korean Hangul represents each square shape separately so requires over 10,000 characters. If it hadn't been like that, Korea wouldn't have accepted Unicode for use because it would have multiplied the number of bytes required to represent characters from their existing system. There's 10,000 Japanese/Chinese Kanji in modern use, plus another less commonly used 65,000 encoded, and there's no consistent way to represent them as components. > If unicode had stuck to representing existing characters and symbols and said no to requests for stuff like emoji and klingon, string representation in modern software could have been kept a whole lot simpler. Klingon was rejected for encoding by the Unicode Consortium in 1997, and remains excluded to this day. emoji was included as a political nod to the Japanese, who even today are the main holdouts to Unicode being the exclusive code for worldwide use.

btn 13y ago

this is perfectly adequate to support the characters from all major languages in use worldwide, plus emoji Emoji characters are in the range U+1F300..U+1F5FF, which is outside the BMP (up to U+FFFF), and will be encoded with surrogate pairs in UTF-16. Also, even within the BMP, assuming one 16-bit UTF-16 block is the same as one "character" is dangerous, as combining characters (for example) should arguably not be separated from their base character. If unicode had stuck to representing existing characters and symbols and said no to requests for stuff like emoji and klingon, string representation in modern software could have been kept a whole lot simpler. Emoji and Klingon are trivial to deal with in comparison to the issues that can appear with "real" languages when you start making assumptions (cf. combining characters, normalisation forms, CJK unification and compatibility blocks, presentation forms, collation, etc.)

vorg 13y ago

> With 65,536 possible combinations, this is perfectly adequate to support the characters from all major languages in use worldwide [1] It depends on how you represent characters. Korean Hangul represents each square shape separately so requires over 10,000 characters. If it hadn't been like that, Korea wouldn't have accepted Unicode for use because it would have multiplied the number of bytes required to represent characters from their existing system. There's 10,000 Japanese/Chinese Kanji in modern use, plus another less commonly used 65,000 encoded, and there's no consistent way to represent them as components. > If unicode had stuck to representing existing characters and symbols and said no to requests for stuff like emoji and klingon, string representation in modern software could have been kept a whole lot simpler. Klingon was rejected for encoding by the Unicode Consortium in 1997, and remains excluded to this day. emoji was included as a political nod to the Japanese, who even today are the main holdouts to Unicode being the exclusive code for worldwide use.