6 ms·

The broader question is, I think, how can any textual sequence cause a crash. Text should be regarded very skeptically by the text renderer. What could this be

The broader question is, I think, how can any textual sequence cause a crash. Text should be regarded very skeptically by the text renderer. What could this be doing that it invokes a crash? Where else is this team not being careful? It reeks of process problems.

spaceywilly 9y ago
My question was, why was there not a unit test for this? It seems like it would be trivial to step through every character combination for each language they support and make sure it doesn't cause a crash.
- TazeTSchnitzel 9y ago
  There are more possible character sequences than there are stars in the sky.
  - jrochkind1 9y ago
    Googling, "If you tabulate all stars visible down to magnitude 6.5, thought to be the faintest stars still visible to the unaided eye, the entire sky contains some 9,000 stars. Since you can only see half the sky at any time, that means there are as many as 4,500 stars visible in your sky tonight." Apparently not a very high bar.
  - tinix 9y ago
    Why does everyone assume only this one specific character combination causes the issue? Generalists can't generalize, I guess. IOS/OSX are notoriously bad at unicode parsing... These have been problems for YEARS and there are many many known character combinations that cause these types of issues. You seriously think every character has unique code to process it? Of course not... there are general subsystems that process all of these things, and those can easily be tested. Literally, copy/pasting the Arabic word for "hummus" (حُمُّص) about 20 times will crash OSX/IOS. It's likely a mixture of RTL and LTR along w/ BOM and other unicode weirdness that causes these issues. That is something that is trivial to test for. For added effect, just shovel ح̷̵̷ُ̧̛̟͙̙̠͖̺̞̟̬͖͙̯̭̞̘͚̻̱͉̦̲̑ͫ͐ͬ̏̽ͨ̐̃̃́ͬ͑̃͛̍̃ͭ̄ͬ̊͗̆̇́͘͜͜͞ م̶̸̸ُّ̢̧̨͓̮̣̺̤̟͓͕̯̯̬͉͍̥̥̹͉͉̠̣̰̜̻͓͖̮̫͎̯͙͇̳͛ͩͨ͒ͦ̏̊͒ͩ̅̅̑ͤ̋ͮͩ̔̒͆̔̂͐ͧ͒̐ͭ̎̕̚͜͜͞͞͠ ص̑ͨͦ̆͗̌̔ͬͬ̈̌̑͏̧̙̰̩̭̜̮̺͚̼̙͉̱̭͉͖̤͞ concatenated a few times to any IOS or OSX application (chat apps work great), and you can crash even the most recent "fixed" versions of IOS/OSX, still, to this day, because these bugs have never been fixed at their core. IOS/OSX unicode support is awful, and will remain awful, if the past has been any indication of the future. An article that claims to "pick apart" the issue, that only mentions "unicode" once in the article body is pretty sad... and no mentions of byte order mark, or endianness. Some dissection...
    - Manishearth 9y ago
      I don't say "unicode" because I'm not certain it has to do specifically with unicode handling, and not a font stack bug (that would occur on other encodings). One of the previous crashes, for example, had nothing to do with unicode -- it had to do with the fact that some arabic strings get larger, visually, when you shorten them. Folks these days say "unicode" when talking about anything relevant to non-Latin text, which munges up the issue, which is why I specifically avoiding saying that word. BOM or endianness don't seem to be relevant to this bug.
      - tinix 9y ago
        The problem is not with the characters themselves though, it's with how CoreText processes and parses the unicode symbols. Don't get me wrong, I know fonts can be malicious, even... but given the history[0] here, with these unicode[1] issues... I think it's pretty safe to say the issue is not with a specific font, per-se, but with CoreText and unicode parsing. For example, when I ran OSX on my old macbook pro, I used open source fonts on my terminal, and this unicode parsing bug still happened. In 2015 an Apple spokesperson had this to say: "We are aware of an iMessage issue caused by a specific series of unicode characters and we will make a fix available in a software update." > BOM or endianness don't seem to be relevant to this bug. Yet on your blog you allude to just that with the left/right comments, though, to be fair, you state that you really don't know the problem: > I don’t really have one guess as to what’s going on here – I’d love to see what people think – but my current guess is that the “affinity” of the virama to the left instead of the right confuses the algorithm that handles ZWNJs after viramas into thinking the ZWNJ applies to the virama (it doesn’t, there’s a consonant in between), and this leads to some numbers not matching up and causing a buffer overflow or something. This is claimed to be a dissection of the issue, but there is not even a stack trace present, and yet you joke about that... > Yes, I could attach a debugger to the crashing process and investigate that instead, but that’s no fun Nah, you should do that... and you'll likely see that it's CoreText being the same old piece of shit as usual. If it was a font problem, then loading that font on a different system that uses a different rendering engine should reproduce the same problem. It doesn't, I tried that years ago. Sure, I would posit, that potentially, this is a whole different bug, but... given the history, and the repeated failed attempts to fix this entire class of issues... it's safe to say that IOS and OSX do not handle unicode very well. [0] https://www.theregister.co.uk/2013/09/04/unicode_of_death_crash/ https://www.theregister.co.uk/2013/09/04/unicode_of_death_cr... [1] https://www.theregister.co.uk/2015/05/27/text_message_unicode_ios_osx_vulnerability/ https://www.theregister.co.uk/2015/05/27/text_message_unicod...
- brian-armstrong 9y ago
  I don't know if I agree with that. The sequence here is something like 5 or 6 characters. There's some tens of thousands of unicode characters, which suggests something like (10000)6 combinations to test. Fuzzing might be more fruitful, but even then I'm skeptical it can find obscure crashes. There's no replacement for defensive programming.
  - jcranmer 9y ago
    Concolic execution as a form of whitebox fuzzing is a very effective way to generate these kinds of crashes, especially if the crash is simple enough that it's the input stream that's generating it, and not positioning the program into a very specific state to get this kind of input string to crash it.
- nikanj 9y ago
  Unicode allows for 17 planes, each of 65,536 possible characters (or 'code points'). This gives a total of 1,114,112. This crashing sequence is five characters long. Your unit tests would have to go through 1.71650179e30 sequences to be guaranteed to catch this one. At a test rate of 1 millisecond per sequence, that's just 4×10^9 × the age of the universe, according to wolfram alpha.
  - jfk13 9y ago
    Rendering Indic scripts with AAT fonts involves a series of finite state machines that are stored in the individual font. So don't forget to multiply by the number of different fonts that each need to be tested.
  - ancarda 9y ago
    Do you know how was this bug found? Are there just enough people using iOS that these sorts of bugs can be found by mistake, or is someone fuzzing CoreText? Perhaps that can be applied to provide some kind of test coverage? Even if it’s not complete?
    - perkee 9y ago
      This sequence begins the Telugu word for "knowledge" so maybe someone texted that to someone and it went viral from there. This is, of course, only speculation.
      - bonzini 9y ago
        Does the word include the zwnj character? How do you input it?
        Manishearth 9y ago
        It does not include the zwnj, that somehow snuck in. Most keyboards don't support directly inputting a zwnj, but may support it for specific combinations. For example my Marathi keyboard supports typing eyelash rephs (e.g. in र्‍क) which includes zwj. However I'm not aware of any such things in Telugu aside from explicit virama-showing which rarely exists in input methods (and doesn't end up with zwnj in the position shown here, but that could have happened after editing).
- nottorp 9y ago
  I bet they have unit tests for all emojis...
  - tinix 9y ago
    this comment is under-rated. haha, you know it's true.... and sad.
throwawayfinal 9y ago
displaying language is full of corner cases. imagine having to render both english, japanese, and arabic in the same code paths, and inline with each other text display code is not so simple as it seems, and don't fall into the trap of "how hard could it really be"
- brian-armstrong 9y ago
  I'm not saying it isn't hard - I acknowledge that getting it right is a huge task. But the cost of failure should be rendering incorrectly, not crashing or memory corruption.
  - jay-anderson 9y ago
    There was a similar bug about a year ago as well. I'm surprised more fuzzing, static analysis, and general testing hasn't gone into this area since there have been issues found there already. Maybe they have done this extra work and just didn't catch this one. It's hard to know. Still feels like the way it fails is unacceptable.
jcranmer 9y ago
Latin text gives people a misleading idea as to how simple text is. For Latin, each character is generally an independent unit with independent metrics isolated from its environment, with a small set of exceptions to this rule (ligatures). East Asian ideographs bring up interesting questions about what constitutes a character, with Unicode "solving" the problem by saying "every distinct rendering is a distinct character," necessitating somewhere in the region of 80,000 characters or so once all of them get added. Even more difficult are scripts like Korean Hangul or Egyptian and Mayan glyphs, which are composed from a relatively small set of independent units but laid out in blocks and sub-blocks themselves composed in linear text. Unicode has both precomposed Hangul characters and the individual Jamo radicals, but it has currently punted on being able to accurately Egyptian or Mayan text in the first place (although they do appear to be revisiting that decision). Scripts like Arabic differ from Latin in that glyphs changing shape according to their surrounding context is the norm rather than the exception. However, in Arabic, you largely break this down into an initial/medial/final form, with some characters inducing a shift from medial back to initial. Indic scripts go far beyond this by needing to treat entire consonant clusters as single rendered glyphs. The end result is that it's very easy to find that invariants that one expects to exist if one is used to Latin text to be violated in other languages. Some concepts of text metrics might not even exist in the first place. As speculated elsewhere, it's probable that the text is crashing because someone insufficiently versed in scripts is asserting an invariant that doesn't actually exist. It does not appear to be a rendering issue, but rather a slightly higher level operation on top of that instead. Finding these sorts of issues pretty much requires extensive fuzzing with known problematic scenarios. The fault is caused by "I didn't know this exists" which is both a very reasonable situation (few UI experts are well-versed in the complexities of foreign scripts) and very hard to solve from a process perspective.
- Manishearth 9y ago
  > Indic scripts go far beyond this by needing to treat entire consonant clusters as single rendered glyphs. FWIW, Arabic does this too, just that most default Arabic fonts don't. https://www.google.com/get/noto/#nastaliq-aran https://www.google.com/get/noto/#nastaliq-aran contains a bunch of specialized ligatures (and is overall a very complicated font). I always say: There's a reason a lot of the folks working on font shaping are Persian/Arabic speakers :)
minimaxir 9y ago
A good example of test sequences causing weird behavior/crashes are RTL languages (like Hebrew and Arabic) and CJK characters (Chinese/Japanese/Korean), which have UI implications as well. While working at Apple in Software QA, both of these cases in part inspired me to create the Big List of Naughty Strings years ago: https://github.com/minimaxir/big-list-of-naughty-strings https://github.com/minimaxir/big-list-of-naughty-strings
- gwern 9y ago
  Are you adding OP's strings to the list?
  - minimaxir 9y ago
    After the crashes are fixed on iOS, sure.

spaceywilly 9y ago

My question was, why was there not a unit test for this? It seems like it would be trivial to step through every character combination for each language they support and make sure it doesn't cause a crash.

TazeTSchnitzel 9y ago
There are more possible character sequences than there are stars in the sky.
- jrochkind1 9y ago
  Googling, "If you tabulate all stars visible down to magnitude 6.5, thought to be the faintest stars still visible to the unaided eye, the entire sky contains some 9,000 stars. Since you can only see half the sky at any time, that means there are as many as 4,500 stars visible in your sky tonight." Apparently not a very high bar.
- tinix 9y ago
  Why does everyone assume only this one specific character combination causes the issue? Generalists can't generalize, I guess. IOS/OSX are notoriously bad at unicode parsing... These have been problems for YEARS and there are many many known character combinations that cause these types of issues. You seriously think every character has unique code to process it? Of course not... there are general subsystems that process all of these things, and those can easily be tested. Literally, copy/pasting the Arabic word for "hummus" (حُمُّص) about 20 times will crash OSX/IOS. It's likely a mixture of RTL and LTR along w/ BOM and other unicode weirdness that causes these issues. That is something that is trivial to test for. For added effect, just shovel ح̷̵̷ُ̧̛̟͙̙̠͖̺̞̟̬͖͙̯̭̞̘͚̻̱͉̦̲̑ͫ͐ͬ̏̽ͨ̐̃̃́ͬ͑̃͛̍̃ͭ̄ͬ̊͗̆̇́͘͜͜͞ م̶̸̸ُّ̢̧̨͓̮̣̺̤̟͓͕̯̯̬͉͍̥̥̹͉͉̠̣̰̜̻͓͖̮̫͎̯͙͇̳͛ͩͨ͒ͦ̏̊͒ͩ̅̅̑ͤ̋ͮͩ̔̒͆̔̂͐ͧ͒̐ͭ̎̕̚͜͜͞͞͠ ص̑ͨͦ̆͗̌̔ͬͬ̈̌̑͏̧̙̰̩̭̜̮̺͚̼̙͉̱̭͉͖̤͞ concatenated a few times to any IOS or OSX application (chat apps work great), and you can crash even the most recent "fixed" versions of IOS/OSX, still, to this day, because these bugs have never been fixed at their core. IOS/OSX unicode support is awful, and will remain awful, if the past has been any indication of the future. An article that claims to "pick apart" the issue, that only mentions "unicode" once in the article body is pretty sad... and no mentions of byte order mark, or endianness. Some dissection...
  - Manishearth 9y ago
    I don't say "unicode" because I'm not certain it has to do specifically with unicode handling, and not a font stack bug (that would occur on other encodings). One of the previous crashes, for example, had nothing to do with unicode -- it had to do with the fact that some arabic strings get larger, visually, when you shorten them. Folks these days say "unicode" when talking about anything relevant to non-Latin text, which munges up the issue, which is why I specifically avoiding saying that word. BOM or endianness don't seem to be relevant to this bug.
    - tinix 9y ago
      The problem is not with the characters themselves though, it's with how CoreText processes and parses the unicode symbols. Don't get me wrong, I know fonts can be malicious, even... but given the history[0] here, with these unicode[1] issues... I think it's pretty safe to say the issue is not with a specific font, per-se, but with CoreText and unicode parsing. For example, when I ran OSX on my old macbook pro, I used open source fonts on my terminal, and this unicode parsing bug still happened. In 2015 an Apple spokesperson had this to say: "We are aware of an iMessage issue caused by a specific series of unicode characters and we will make a fix available in a software update." > BOM or endianness don't seem to be relevant to this bug. Yet on your blog you allude to just that with the left/right comments, though, to be fair, you state that you really don't know the problem: > I don’t really have one guess as to what’s going on here – I’d love to see what people think – but my current guess is that the “affinity” of the virama to the left instead of the right confuses the algorithm that handles ZWNJs after viramas into thinking the ZWNJ applies to the virama (it doesn’t, there’s a consonant in between), and this leads to some numbers not matching up and causing a buffer overflow or something. This is claimed to be a dissection of the issue, but there is not even a stack trace present, and yet you joke about that... > Yes, I could attach a debugger to the crashing process and investigate that instead, but that’s no fun Nah, you should do that... and you'll likely see that it's CoreText being the same old piece of shit as usual. If it was a font problem, then loading that font on a different system that uses a different rendering engine should reproduce the same problem. It doesn't, I tried that years ago. Sure, I would posit, that potentially, this is a whole different bug, but... given the history, and the repeated failed attempts to fix this entire class of issues... it's safe to say that IOS and OSX do not handle unicode very well. [0] https://www.theregister.co.uk/2013/09/04/unicode_of_death_crash/ https://www.theregister.co.uk/2013/09/04/unicode_of_death_cr... [1] https://www.theregister.co.uk/2015/05/27/text_message_unicode_ios_osx_vulnerability/ https://www.theregister.co.uk/2015/05/27/text_message_unicod...
brian-armstrong 9y ago
I don't know if I agree with that. The sequence here is something like 5 or 6 characters. There's some tens of thousands of unicode characters, which suggests something like (10000)6 combinations to test. Fuzzing might be more fruitful, but even then I'm skeptical it can find obscure crashes. There's no replacement for defensive programming.
- jcranmer 9y ago
  Concolic execution as a form of whitebox fuzzing is a very effective way to generate these kinds of crashes, especially if the crash is simple enough that it's the input stream that's generating it, and not positioning the program into a very specific state to get this kind of input string to crash it.
nikanj 9y ago
Unicode allows for 17 planes, each of 65,536 possible characters (or 'code points'). This gives a total of 1,114,112. This crashing sequence is five characters long. Your unit tests would have to go through 1.71650179e30 sequences to be guaranteed to catch this one. At a test rate of 1 millisecond per sequence, that's just 4×10^9 × the age of the universe, according to wolfram alpha.
- jfk13 9y ago
  Rendering Indic scripts with AAT fonts involves a series of finite state machines that are stored in the individual font. So don't forget to multiply by the number of different fonts that each need to be tested.
- ancarda 9y ago
  Do you know how was this bug found? Are there just enough people using iOS that these sorts of bugs can be found by mistake, or is someone fuzzing CoreText? Perhaps that can be applied to provide some kind of test coverage? Even if it’s not complete?
  - perkee 9y ago
    This sequence begins the Telugu word for "knowledge" so maybe someone texted that to someone and it went viral from there. This is, of course, only speculation.
    - bonzini 9y ago
      Does the word include the zwnj character? How do you input it?
      - Manishearth 9y ago
        It does not include the zwnj, that somehow snuck in. Most keyboards don't support directly inputting a zwnj, but may support it for specific combinations. For example my Marathi keyboard supports typing eyelash rephs (e.g. in र्‍क) which includes zwj. However I'm not aware of any such things in Telugu aside from explicit virama-showing which rarely exists in input methods (and doesn't end up with zwnj in the position shown here, but that could have happened after editing).
nottorp 9y ago
I bet they have unit tests for all emojis...
- tinix 9y ago
  this comment is under-rated. haha, you know it's true.... and sad.

throwawayfinal 9y ago

displaying language is full of corner cases. imagine having to render both english, japanese, and arabic in the same code paths, and inline with each other text display code is not so simple as it seems, and don't fall into the trap of "how hard could it really be"

brian-armstrong 9y ago
I'm not saying it isn't hard - I acknowledge that getting it right is a huge task. But the cost of failure should be rendering incorrectly, not crashing or memory corruption.
- jay-anderson 9y ago
  There was a similar bug about a year ago as well. I'm surprised more fuzzing, static analysis, and general testing hasn't gone into this area since there have been issues found there already. Maybe they have done this extra work and just didn't catch this one. It's hard to know. Still feels like the way it fails is unacceptable.

jcranmer 9y ago

Latin text gives people a misleading idea as to how simple text is. For Latin, each character is generally an independent unit with independent metrics isolated from its environment, with a small set of exceptions to this rule (ligatures). East Asian ideographs bring up interesting questions about what constitutes a character, with Unicode "solving" the problem by saying "every distinct rendering is a distinct character," necessitating somewhere in the region of 80,000 characters or so once all of them get added. Even more difficult are scripts like Korean Hangul or Egyptian and Mayan glyphs, which are composed from a relatively small set of independent units but laid out in blocks and sub-blocks themselves composed in linear text. Unicode has both precomposed Hangul characters and the individual Jamo radicals, but it has currently punted on being able to accurately Egyptian or Mayan text in the first place (although they do appear to be revisiting that decision). Scripts like Arabic differ from Latin in that glyphs changing shape according to their surrounding context is the norm rather than the exception. However, in Arabic, you largely break this down into an initial/medial/final form, with some characters inducing a shift from medial back to initial. Indic scripts go far beyond this by needing to treat entire consonant clusters as single rendered glyphs. The end result is that it's very easy to find that invariants that one expects to exist if one is used to Latin text to be violated in other languages. Some concepts of text metrics might not even exist in the first place. As speculated elsewhere, it's probable that the text is crashing because someone insufficiently versed in scripts is asserting an invariant that doesn't actually exist. It does not appear to be a rendering issue, but rather a slightly higher level operation on top of that instead. Finding these sorts of issues pretty much requires extensive fuzzing with known problematic scenarios. The fault is caused by "I didn't know this exists" which is both a very reasonable situation (few UI experts are well-versed in the complexities of foreign scripts) and very hard to solve from a process perspective.

Manishearth 9y ago
> Indic scripts go far beyond this by needing to treat entire consonant clusters as single rendered glyphs. FWIW, Arabic does this too, just that most default Arabic fonts don't. https://www.google.com/get/noto/#nastaliq-aran https://www.google.com/get/noto/#nastaliq-aran contains a bunch of specialized ligatures (and is overall a very complicated font). I always say: There's a reason a lot of the folks working on font shaping are Persian/Arabic speakers :)

minimaxir 9y ago

A good example of test sequences causing weird behavior/crashes are RTL languages (like Hebrew and Arabic) and CJK characters (Chinese/Japanese/Korean), which have UI implications as well. While working at Apple in Software QA, both of these cases in part inspired me to create the Big List of Naughty Strings years ago: https://github.com/minimaxir/big-list-of-naughty-strings https://github.com/minimaxir/big-list-of-naughty-strings

gwern 9y ago
Are you adding OP's strings to the list?
- minimaxir 9y ago
  After the crashes are fixed on iOS, sure.