16 ms·

Picking Apart the Crashing iOS String

457 points by Manishearth 9y ago

donarb 9y ago
Very interesting article. It gives a bit more perspective to the problem than the simplistic view that iOS has a problem displaying a certain character onscreen. This particular bug shows the problem with multibyte characters (terminology may be wrong, perhaps multi glyph is better) where certain parts of the character become left or right associative based on context.
- Manishearth 9y ago
  You want "multi-codepoint graphemes" probably. Or really "things involving combining characters". There isn't really a term for this.
  - bitwize 9y ago
    I believe the Unicode term is "grapheme cluster".
    - Manishearth 9y ago
      Not exactly, no. क्र is currently not a grapheme cluster, for example (there is a proposed change to UAX 29 to make it such, but that doesn't handle all such cases). The strings in this situation are three grapheme clusters according to the spec even though they usually render as a single "thingy".
brian-armstrong 9y ago
The broader question is, I think, how can any textual sequence cause a crash. Text should be regarded very skeptically by the text renderer. What could this be doing that it invokes a crash? Where else is this team not being careful? It reeks of process problems.
- spaceywilly 9y ago
  My question was, why was there not a unit test for this? It seems like it would be trivial to step through every character combination for each language they support and make sure it doesn't cause a crash.
  - TazeTSchnitzel 9y ago
    There are more possible character sequences than there are stars in the sky.
    - jrochkind1 9y ago
      Googling, "If you tabulate all stars visible down to magnitude 6.5, thought to be the faintest stars still visible to the unaided eye, the entire sky contains some 9,000 stars. Since you can only see half the sky at any time, that means there are as many as 4,500 stars visible in your sky tonight." Apparently not a very high bar.
    - tinix 9y ago
      Why does everyone assume only this one specific character combination causes the issue? Generalists can't generalize, I guess. IOS/OSX are notoriously bad at unicode parsing... These have been problems for YEARS and there are many many known character combinations that cause these types of issues. You seriously think every character has unique code to process it? Of course not... there are general subsystems that process all of these things, and those can easily be tested. Literally, copy/pasting the Arabic word for "hummus" (حُمُّص) about 20 times will crash OSX/IOS. It's likely a mixture of RTL and LTR along w/ BOM and other unicode weirdness that causes these issues. That is something that is trivial to test for. For added effect, just shovel ح̷̵̷ُ̧̛̟͙̙̠͖̺̞̟̬͖͙̯̭̞̘͚̻̱͉̦̲̑ͫ͐ͬ̏̽ͨ̐̃̃́ͬ͑̃͛̍̃ͭ̄ͬ̊͗̆̇́͘͜͜͞ م̶̸̸ُّ̢̧̨͓̮̣̺̤̟͓͕̯̯̬͉͍̥̥̹͉͉̠̣̰̜̻͓͖̮̫͎̯͙͇̳͛ͩͨ͒ͦ̏̊͒ͩ̅̅̑ͤ̋ͮͩ̔̒͆̔̂͐ͧ͒̐ͭ̎̕̚͜͜͞͞͠ ص̑ͨͦ̆͗̌̔ͬͬ̈̌̑͏̧̙̰̩̭̜̮̺͚̼̙͉̱̭͉͖̤͞ concatenated a few times to any IOS or OSX application (chat apps work great), and you can crash even the most recent "fixed" versions of IOS/OSX, still, to this day, because these bugs have never been fixed at their core. IOS/OSX unicode support is awful, and will remain awful, if the past has been any indication of the future. An article that claims to "pick apart" the issue, that only mentions "unicode" once in the article body is pretty sad... and no mentions of byte order mark, or endianness. Some dissection...
Diggsey 9y ago
My guess would be that it's some aspect of measuring the text that is causing the crash: when you click in an editable text box, there is code to track down where the cursor should be placed. This is done by measuring various sub-strings of the whole line. If measuring the sub-strings gives surprising results (sub-strings being visibly longer for example), this could cause the algorithm to fail in any number of interesting ways: for example if a binary search is used to locate the cursor position, it could break the invariants of the binary search.
- goalieca 9y ago
  I believe the renderer is a brand new metal 2 for high sierra. The crashing 3rd party apps for iOS are using this new rendering too?
  - jfk13 9y ago
    It can crash on (non-high) Sierra, too. The crash is in Core Text shaping, rather than in actual painting, so somewhat above the metal level IIUC.
- jcranmer 9y ago
  I'd have to dump strings as grapheme clusters, but it's quite possible that there's some grapheme cluster weirdness going on here.
- Manishearth 9y ago
  Well, the crash occurs for Spotlight without me clicking anything or having any cursors anywhere. But yeah, this is one of my theories about it. One of the previous crashes had to do with an Arabic string which got longer when you truncated it, which made snipping it to display in a notification have bugs. It's interesting to see it's causing a segfault, i'd expect measuring bugs to cause clean assertions or shitty rendering. Which is why I'm also wondering if it's actually a disagreement on the number of "characters" in the rendered things. > If measuring the sub-strings gives surprising results (sub-strings being visibly longer for example), this could cause the algorithm to fail in any number of interesting ways: for example if a binary search is used to locate the cursor position, it could break the invariants of the binary search. Cursor positions are based off of grapheme clusters -- there's a defined algorithm for that. Though different parts of the system may disagree on the specifics of the algorithm causing such a crash. However, that doesn't gel with the fact that it's only specific consonants causing this, all versions of UAX 29 do not consider any differences between Indic consonants for a single given script.
kibwen 9y ago
Has anyone yet traced this to a specific syscall? Seems like the perfect opportunity for whipping up a fuzzer to spit arbitrary unicode into the system and see what else crashes.
- Sir_Cmpwn 9y ago
  I doubt very seriously that a misbehaving syscall is involved in the crash.
- enzanki_ars 9y ago
  Would that even be possible on iOS for non Apple employees? My idea would be just to create a program that continually creates Unicode strings and display them on screen with a pause before creating and displaying the next. Record the screen and see if any thing exciting happens. Would never end, if there was a SETI @ Home style program for it with insentives for running it would be cool.
  - thephyber 9y ago
    > Would that even be possible on iOS for non Apple employees? iOS emulator (part of XCode) might be enough.
    - enzanki_ars 9y ago
      Ah. Forgot that existed for some reason. (Windows/Linux user that gets screwed out of programming for my own phone. Hate that....)
      - gsich 9y ago
        Use Hackintosh or a VM.
        enzanki_ars 9y ago
        Given that Apple does not design their OS to work on anything but their hardware, the pain I would incur trying to set up either option is just not worth it. Unless someone has published a stable and updated pre-built VM for macOS, it just seems like nothing but trouble.
jrochkind1 9y ago
What I'm really curious about is how bugs/errors in the iOS typesetting algorithm result in a crash, rather than just wrong or nonsense typesetting -- and how the last time this happened, they appearently just fixed the specific case, but not the ability of bugs/errors in the typesetting algorithm to crash the system. I am not surprised there will be errors/bugs in the typesetting algorithm, as the OP demonstrates, this stuff is extraordinarily complicated to do for every language/all of unicode. But if you have something so complicated that bugs/errors aren't surprising -- you'd think you'd want to make sure they didn't cause hard crashes.
- stcredzero 9y ago
  you'd think you'd want to make sure they didn't cause hard crashes. Any Apple people want to chime in? Why wasn't the general bug -- where there could be even the possibility of a "text crash" -- fixed?
  - glibgil 9y ago
    If Apple people chime in I think they get beheaded. It never happens
    - saagarjha 9y ago
      They do sometimes, but generally not for topics such as these.
    - acheron 9y ago
      “The bomb says no, Brian.” https://www.penny-arcade.com/comic/2007/05/23/ https://www.penny-arcade.com/comic/2007/05/23/
    - AceJohnny2 9y ago
      Absolutely. It's drilled and repeated on a regular schedule: Shut the fuck up.
  - ghusbands 9y ago
    You realize that there's no such thing as "don't ever crash" fix, right? Maybe they added some defensive code and maybe they didn't, but if they're using an unsafe language, there's always the possibility of more such issues.
gatkinso 9y ago
shoulda used JS
kens 9y ago
Not directly related to the crash, but I have a question about Telugu and similar scripts: How do their speakers think about the structure of the script? Do they consider each vowel and consonant a separate "thing" that just happens to get written as a complex grapheme, or is the grapheme the unit you think about and it just happens to be made up of smaller parts? I.e https://en.wikipedia.org/wiki/Telugu_script#Consonant_Conjuncts https://en.wikipedia.org/wiki/Telugu_script#Consonant_Conjun... Also, do you learn the large number of graphemes separately? Or are they the "obvious" way to write the consonant and vowel? Or is there a set of rules you learn?
- jsjohnst 9y ago
  I’m curious too. Koreans, for example, think in an alphabet similar to ours (actually fewer letters), but it forms into a grapheme. I don’t think that is the case here, but don’t know for sure.
  - Manishearth 9y ago
    Not Korean, but the Korean case is different, a bit. Korean syllable blocks are viewed as syllable blocks -- a new concept, whereas Indic consonant clusters are simultaneously viewed as letters of their own and fundamentally composed of more letters.
- Manishearth 9y ago
  I can answer your question for Devanagari: - You normally think of clusters as single "letters". The word for them in Marathi is "joined letter". This notion of "letter" may not be totally in line with what English speakers would expect. - You still think of clusters as having component parts that make it up. However, क्ष/ज्ञ and sometimes त्र are thought of as their own "fundamental" consonants even though they're clusters. I think folks think of things like क्र as being both -- a letter of its own, that is built up from component letters. - Most of the clusters in Devanagari are predictable. So for any given pair you usually know how to make a "half form" for the first one and join it on, or you know how to stack them. Usually whilst reading it's obvious because you see the components; and whilst writing it's usually fine to try what makes sense. - For unpredictable clusters, you just eventually get used to them. No different from learning that some words are spelled weirdly in English, really. Though I very much dislike ह्म , ह्य, and द्य since they're kinda confusing. I think Bengali has a lot more unpredictable conjuncts; I stumbled across this a lot when I tried to learn the script a year ago. I suspect folks just get used to those.
blackflame7000 9y ago
Does anyone else think that this could be related to Auto-Correct having problems with the malformed expressions?
- Manishearth 9y ago
  This happens whenever the system is asked to render text; it doesn't have to be editable. (it's just easier to trigger in text boxes because you paste and that's it)
  - blackflame7000 9y ago
    Oh, gotcha. Yea not sure auto-correct could cause that. I initially read it was only related to text-entry and thought auto-correct was likely since it was affecting multiple apps indicating an OS issue.
komali2 9y ago
Not sure why I'm not able to trigger this crash on any Apple devices in the office - saw on twitter about how it was messing with iphones even if it was an SSID, unable to replicate here though.
foota 9y ago
Funny, this is the same person that wrote the new str find functionality in rust 1.24
- Manishearth 9y ago
  Turns out I like doing unicodey things :)
ahstilde 9y ago
People interested in this bug may also be interested in "An Introduction to Writing Systems & Unicode." Part 3[1] in particular discusses complex scripts. [1] https://r12a.github.io/scripts/tutorial/part3 https://r12a.github.io/scripts/tutorial/part3
torstenvl 9y ago
I cannot trigger this crash on iOS 10.3.2, despite repeated attempts. I can reliably crash friends' phones by iMessage for all friends on iOS 11. That suggests to me that Apple made changes to CoreText, and did not perform adequate regression testing. All software has bugs. I understand that. But I suspect a large part of power users' and developers' growing frustration with Apple is that they keep introducing severe, kernel-panicking, root-exposing bugs in software that previously did not exhibit the problematic behavior. Honestly, how do you not have a stringent regression testing requirement for changes to the "Core" of the operating system?
- arvinsim 9y ago
  > Honestly, how do you not have a stringent regression testing requirement for changes to the "Core" of the operating system? By not managing expectations and deadlines properly. Especially if schedules are overriden from a manager who is not aware of the ramifications of changing something in the core.
  - zatkin 9y ago
    And this is about to change, according to some recent news: https://www.bloomberg.com/news/articles/2018-02-12/how-apple-plans-to-root-out-bugs-revamp-iphone-software https://www.bloomberg.com/news/articles/2018-02-12/how-apple...
- paulriddle 9y ago
  Well. Is it really bad to have bugs though? I think it can be good, it's free advertising it gets people's attention for some time. Which is good. Even being hacked is good, you can always turn it into PR. Maybe Apple doesn't need this as much, but 99.9% of other companies do. Perhaps even, security consultancies should provide a new kind of service "fake hacking", which is really a PR campaign. You introduce a bug which is kind of severe but not really detrimental. It's better if it allows to compromise a user. Publish tutorials, videos, tweet about it, create panic, make it cool to abuse the system. Give it a name, create a dedicated web page. And then you squash the bug, you fix it. People will remember that you fix stuff, so your company and product must be good because you care.
svat 9y ago
(I should probably try this myself, but…) The equivalent programs on other systems are HarfBuzz (open-source) and Uniscribe / DirectWrite (Windows). Did you consider checking what they do with the equivalent text (presumably they don't crash)?
- mappu 9y ago
  No crashes on Debian harfbuzz.
calvinbhai 9y ago
@Manishearth Thats a great explanation (though quite a bit of it was too much for me to understand). The only correlation I can think of, is that the Kannada script keyboard and full support for the script rendering in the UI didn't exist in iOS until iOS 11.0. So it is crashing for Indic scripts that were supported pre iOS 11. Probably Apple tweaked something in iOS 11 for the languages already supported and broke something.
- Manishearth 9y ago
  Huh, that's an interesting theory.
ollin 9y ago
Does anyone have a link to a thorough reverse-engineering of the bug from someone decompiling/source-diving? I tried replicating it on High Sierra out of curiosity (single text file with the broken character, opened in TextEdit to cause a crash) and noticed that the crash doesn't happen in consistently the same way each time (I got 8 crashes in a calloc call in CoreText's OTL::CoverageBitmap::Reset, 4 crashes in a malloc in Foundation with no CoreText in the stack trace, plus 10 KERN_INVALID_ADDRESS errors from various places, as well as one time it opened successfully with no immediate problems). Seems like a memory corruption error somewhere earlier, but I'm not sure how to figure out where–I tried rooting around in CoreText with Hopper for a bit but didn't get very far.
- Manishearth 9y ago
  http://www.cloudscan.me/2018/02/coretext-crash-osx-10133-lldb-voltron.html http://www.cloudscan.me/2018/02/coretext-crash-osx-10133-lld...
0x0 9y ago
Fonts are hard. I don't think CoreText is worse than others, most vendors seem to have a history of horrors, and at least it's not rendering fonts in a kernel driver like Windows did(!?) - see for example the Project Zero series of "one font vulnerability to rule them all" - https://googleprojectzero.blogspot.com/2015/07/one-font-vulnerability-to-rule-them-all.html https://googleprojectzero.blogspot.com/2015/07/one-font-vuln...
chris_wot 9y ago
Would fuzzing help with this?
bawana 9y ago
I went to the Apple Store in Burlington, MA yesterday (2/16/2018) None of the 'geniuses' there knew about this bug or how to fix it. They reminded me of microsoft drones following a troubleshooting script. 1) Reboot the phone 2) reset the phone 3) do a dfu restore.. After they did this my phone would not recognize its sim card. You see my phone originally had AT&T service. Then I switched to sprint some months back. I guess the dfu restore ended up relocking my phone to AT&T. At least that's what the Apple geniuses said. Then they said I had to go to a sprint store. This is the first time I was disappointed with Apple's human help. I remember past experiences going in there with a frayed power supply cord - they gave me a new one. On other occasions they replaced my phone when they couldn't fix it. They never spent more than 30 mins trying to fix a problem and always fixed it. Their knowledge was deep. This time their opinions were confabulatory - for example one guy said 'your phone does not have the antennae to work with sprint' EVEN THOUGH I reminded him I walked into the store with a working iPhone using sprint. Another one said that sprint jailbroke my phone and that's how they got it to work. WRONG. Another one said that the phone was 'soft unlocked' and relocked itself after the DFU restore. I felt sad for Apple because I know that's not what they were about. Then I suggested they replace my phone, the manager lady's eyes bugged out, her voice became stern and said ' There is no scenario where you walk out of here with a different phone'. This is the first time they wasted 5 hours of my time and actually caused me to have more trouble than what I walked in with. Galaxy here I come. or maybe pixel2

457 points by Manishearth 9y ago

Very interesting article. It gives a bit more perspective to the problem than the simplistic view that iOS has a problem displaying a certain character onscreen. This particular bug shows the problem with multibyte characters (terminology may be wrong, perhaps multi glyph is better) where certain parts of the character become left or right associative based on context.

Manishearth 9y ago
You want "multi-codepoint graphemes" probably. Or really "things involving combining characters". There isn't really a term for this.
- bitwize 9y ago
  I believe the Unicode term is "grapheme cluster".
  - Manishearth 9y ago
    Not exactly, no. क्र is currently not a grapheme cluster, for example (there is a proposed change to UAX 29 to make it such, but that doesn't handle all such cases). The strings in this situation are three grapheme clusters according to the spec even though they usually render as a single "thingy".

brian-armstrong 9y ago

The broader question is, I think, how can any textual sequence cause a crash. Text should be regarded very skeptically by the text renderer. What could this be doing that it invokes a crash? Where else is this team not being careful? It reeks of process problems.

spaceywilly 9y ago
My question was, why was there not a unit test for this? It seems like it would be trivial to step through every character combination for each language they support and make sure it doesn't cause a crash.
- TazeTSchnitzel 9y ago
  There are more possible character sequences than there are stars in the sky.
  - jrochkind1 9y ago
    Googling, "If you tabulate all stars visible down to magnitude 6.5, thought to be the faintest stars still visible to the unaided eye, the entire sky contains some 9,000 stars. Since you can only see half the sky at any time, that means there are as many as 4,500 stars visible in your sky tonight." Apparently not a very high bar.
  - tinix 9y ago
    Why does everyone assume only this one specific character combination causes the issue? Generalists can't generalize, I guess. IOS/OSX are notoriously bad at unicode parsing... These have been problems for YEARS and there are many many known character combinations that cause these types of issues. You seriously think every character has unique code to process it? Of course not... there are general subsystems that process all of these things, and those can easily be tested. Literally, copy/pasting the Arabic word for "hummus" (حُمُّص) about 20 times will crash OSX/IOS. It's likely a mixture of RTL and LTR along w/ BOM and other unicode weirdness that causes these issues. That is something that is trivial to test for. For added effect, just shovel ح̷̵̷ُ̧̛̟͙̙̠͖̺̞̟̬͖͙̯̭̞̘͚̻̱͉̦̲̑ͫ͐ͬ̏̽ͨ̐̃̃́ͬ͑̃͛̍̃ͭ̄ͬ̊͗̆̇́͘͜͜͞ م̶̸̸ُّ̢̧̨͓̮̣̺̤̟͓͕̯̯̬͉͍̥̥̹͉͉̠̣̰̜̻͓͖̮̫͎̯͙͇̳͛ͩͨ͒ͦ̏̊͒ͩ̅̅̑ͤ̋ͮͩ̔̒͆̔̂͐ͧ͒̐ͭ̎̕̚͜͜͞͞͠ ص̑ͨͦ̆͗̌̔ͬͬ̈̌̑͏̧̙̰̩̭̜̮̺͚̼̙͉̱̭͉͖̤͞ concatenated a few times to any IOS or OSX application (chat apps work great), and you can crash even the most recent "fixed" versions of IOS/OSX, still, to this day, because these bugs have never been fixed at their core. IOS/OSX unicode support is awful, and will remain awful, if the past has been any indication of the future. An article that claims to "pick apart" the issue, that only mentions "unicode" once in the article body is pretty sad... and no mentions of byte order mark, or endianness. Some dissection...

Diggsey 9y ago

My guess would be that it's some aspect of measuring the text that is causing the crash: when you click in an editable text box, there is code to track down where the cursor should be placed. This is done by measuring various sub-strings of the whole line. If measuring the sub-strings gives surprising results (sub-strings being visibly longer for example), this could cause the algorithm to fail in any number of interesting ways: for example if a binary search is used to locate the cursor position, it could break the invariants of the binary search.

goalieca 9y ago
I believe the renderer is a brand new metal 2 for high sierra. The crashing 3rd party apps for iOS are using this new rendering too?
- jfk13 9y ago
  It can crash on (non-high) Sierra, too. The crash is in Core Text shaping, rather than in actual painting, so somewhat above the metal level IIUC.
jcranmer 9y ago
I'd have to dump strings as grapheme clusters, but it's quite possible that there's some grapheme cluster weirdness going on here.
Manishearth 9y ago
Well, the crash occurs for Spotlight without me clicking anything or having any cursors anywhere. But yeah, this is one of my theories about it. One of the previous crashes had to do with an Arabic string which got longer when you truncated it, which made snipping it to display in a notification have bugs. It's interesting to see it's causing a segfault, i'd expect measuring bugs to cause clean assertions or shitty rendering. Which is why I'm also wondering if it's actually a disagreement on the number of "characters" in the rendered things. > If measuring the sub-strings gives surprising results (sub-strings being visibly longer for example), this could cause the algorithm to fail in any number of interesting ways: for example if a binary search is used to locate the cursor position, it could break the invariants of the binary search. Cursor positions are based off of grapheme clusters -- there's a defined algorithm for that. Though different parts of the system may disagree on the specifics of the algorithm causing such a crash. However, that doesn't gel with the fact that it's only specific consonants causing this, all versions of UAX 29 do not consider any differences between Indic consonants for a single given script.

kibwen 9y ago

Has anyone yet traced this to a specific syscall? Seems like the perfect opportunity for whipping up a fuzzer to spit arbitrary unicode into the system and see what else crashes.

Sir_Cmpwn 9y ago
I doubt very seriously that a misbehaving syscall is involved in the crash.
enzanki_ars 9y ago
Would that even be possible on iOS for non Apple employees? My idea would be just to create a program that continually creates Unicode strings and display them on screen with a pause before creating and displaying the next. Record the screen and see if any thing exciting happens. Would never end, if there was a SETI @ Home style program for it with insentives for running it would be cool.
- thephyber 9y ago
  > Would that even be possible on iOS for non Apple employees? iOS emulator (part of XCode) might be enough.
  - enzanki_ars 9y ago
    Ah. Forgot that existed for some reason. (Windows/Linux user that gets screwed out of programming for my own phone. Hate that....)
    - gsich 9y ago
      Use Hackintosh or a VM.
      - enzanki_ars 9y ago
        Given that Apple does not design their OS to work on anything but their hardware, the pain I would incur trying to set up either option is just not worth it. Unless someone has published a stable and updated pre-built VM for macOS, it just seems like nothing but trouble.

jrochkind1 9y ago

What I'm really curious about is how bugs/errors in the iOS typesetting algorithm result in a crash, rather than just wrong or nonsense typesetting -- and how the last time this happened, they appearently just fixed the specific case, but not the ability of bugs/errors in the typesetting algorithm to crash the system. I am not surprised there will be errors/bugs in the typesetting algorithm, as the OP demonstrates, this stuff is extraordinarily complicated to do for every language/all of unicode. But if you have something so complicated that bugs/errors aren't surprising -- you'd think you'd want to make sure they didn't cause hard crashes.

stcredzero 9y ago
you'd think you'd want to make sure they didn't cause hard crashes. Any Apple people want to chime in? Why wasn't the general bug -- where there could be even the possibility of a "text crash" -- fixed?
- glibgil 9y ago
  If Apple people chime in I think they get beheaded. It never happens
  - saagarjha 9y ago
    They do sometimes, but generally not for topics such as these.
  - acheron 9y ago
    “The bomb says no, Brian.” https://www.penny-arcade.com/comic/2007/05/23/ https://www.penny-arcade.com/comic/2007/05/23/
  - AceJohnny2 9y ago
    Absolutely. It's drilled and repeated on a regular schedule: Shut the fuck up.
- ghusbands 9y ago
  You realize that there's no such thing as "don't ever crash" fix, right? Maybe they added some defensive code and maybe they didn't, but if they're using an unsafe language, there's always the possibility of more such issues.

gatkinso 9y ago

shoulda used JS

kens 9y ago

Not directly related to the crash, but I have a question about Telugu and similar scripts: How do their speakers think about the structure of the script? Do they consider each vowel and consonant a separate "thing" that just happens to get written as a complex grapheme, or is the grapheme the unit you think about and it just happens to be made up of smaller parts? I.e https://en.wikipedia.org/wiki/Telugu_script#Consonant_Conjuncts https://en.wikipedia.org/wiki/Telugu_script#Consonant_Conjun... Also, do you learn the large number of graphemes separately? Or are they the "obvious" way to write the consonant and vowel? Or is there a set of rules you learn?

jsjohnst 9y ago
I’m curious too. Koreans, for example, think in an alphabet similar to ours (actually fewer letters), but it forms into a grapheme. I don’t think that is the case here, but don’t know for sure.
- Manishearth 9y ago
  Not Korean, but the Korean case is different, a bit. Korean syllable blocks are viewed as syllable blocks -- a new concept, whereas Indic consonant clusters are simultaneously viewed as letters of their own and fundamentally composed of more letters.
Manishearth 9y ago
I can answer your question for Devanagari: - You normally think of clusters as single "letters". The word for them in Marathi is "joined letter". This notion of "letter" may not be totally in line with what English speakers would expect. - You still think of clusters as having component parts that make it up. However, क्ष/ज्ञ and sometimes त्र are thought of as their own "fundamental" consonants even though they're clusters. I think folks think of things like क्र as being both -- a letter of its own, that is built up from component letters. - Most of the clusters in Devanagari are predictable. So for any given pair you usually know how to make a "half form" for the first one and join it on, or you know how to stack them. Usually whilst reading it's obvious because you see the components; and whilst writing it's usually fine to try what makes sense. - For unpredictable clusters, you just eventually get used to them. No different from learning that some words are spelled weirdly in English, really. Though I very much dislike ह्म , ह्य, and द्य since they're kinda confusing. I think Bengali has a lot more unpredictable conjuncts; I stumbled across this a lot when I tried to learn the script a year ago. I suspect folks just get used to those.

blackflame7000 9y ago

Does anyone else think that this could be related to Auto-Correct having problems with the malformed expressions?

Manishearth 9y ago
This happens whenever the system is asked to render text; it doesn't have to be editable. (it's just easier to trigger in text boxes because you paste and that's it)
- blackflame7000 9y ago
  Oh, gotcha. Yea not sure auto-correct could cause that. I initially read it was only related to text-entry and thought auto-correct was likely since it was affecting multiple apps indicating an OS issue.

komali2 9y ago

Not sure why I'm not able to trigger this crash on any Apple devices in the office - saw on twitter about how it was messing with iphones even if it was an SSID, unable to replicate here though.

foota 9y ago

Funny, this is the same person that wrote the new str find functionality in rust 1.24

Manishearth 9y ago
Turns out I like doing unicodey things :)

ahstilde 9y ago

People interested in this bug may also be interested in "An Introduction to Writing Systems & Unicode." Part 3[1] in particular discusses complex scripts. [1] https://r12a.github.io/scripts/tutorial/part3 https://r12a.github.io/scripts/tutorial/part3

torstenvl 9y ago

I cannot trigger this crash on iOS 10.3.2, despite repeated attempts. I can reliably crash friends' phones by iMessage for all friends on iOS 11. That suggests to me that Apple made changes to CoreText, and did not perform adequate regression testing. All software has bugs. I understand that. But I suspect a large part of power users' and developers' growing frustration with Apple is that they keep introducing severe, kernel-panicking, root-exposing bugs in software that previously did not exhibit the problematic behavior. Honestly, how do you not have a stringent regression testing requirement for changes to the "Core" of the operating system?

arvinsim 9y ago
> Honestly, how do you not have a stringent regression testing requirement for changes to the "Core" of the operating system? By not managing expectations and deadlines properly. Especially if schedules are overriden from a manager who is not aware of the ramifications of changing something in the core.
- zatkin 9y ago
  And this is about to change, according to some recent news: https://www.bloomberg.com/news/articles/2018-02-12/how-apple-plans-to-root-out-bugs-revamp-iphone-software https://www.bloomberg.com/news/articles/2018-02-12/how-apple...
paulriddle 9y ago
Well. Is it really bad to have bugs though? I think it can be good, it's free advertising it gets people's attention for some time. Which is good. Even being hacked is good, you can always turn it into PR. Maybe Apple doesn't need this as much, but 99.9% of other companies do. Perhaps even, security consultancies should provide a new kind of service "fake hacking", which is really a PR campaign. You introduce a bug which is kind of severe but not really detrimental. It's better if it allows to compromise a user. Publish tutorials, videos, tweet about it, create panic, make it cool to abuse the system. Give it a name, create a dedicated web page. And then you squash the bug, you fix it. People will remember that you fix stuff, so your company and product must be good because you care.

svat 9y ago

(I should probably try this myself, but…) The equivalent programs on other systems are HarfBuzz (open-source) and Uniscribe / DirectWrite (Windows). Did you consider checking what they do with the equivalent text (presumably they don't crash)?

mappu 9y ago
No crashes on Debian harfbuzz.

calvinbhai 9y ago

@Manishearth Thats a great explanation (though quite a bit of it was too much for me to understand). The only correlation I can think of, is that the Kannada script keyboard and full support for the script rendering in the UI didn't exist in iOS until iOS 11.0. So it is crashing for Indic scripts that were supported pre iOS 11. Probably Apple tweaked something in iOS 11 for the languages already supported and broke something.

Manishearth 9y ago
Huh, that's an interesting theory.

ollin 9y ago

Does anyone have a link to a thorough reverse-engineering of the bug from someone decompiling/source-diving? I tried replicating it on High Sierra out of curiosity (single text file with the broken character, opened in TextEdit to cause a crash) and noticed that the crash doesn't happen in consistently the same way each time (I got 8 crashes in a calloc call in CoreText's OTL::CoverageBitmap::Reset, 4 crashes in a malloc in Foundation with no CoreText in the stack trace, plus 10 KERN_INVALID_ADDRESS errors from various places, as well as one time it opened successfully with no immediate problems). Seems like a memory corruption error somewhere earlier, but I'm not sure how to figure out where–I tried rooting around in CoreText with Hopper for a bit but didn't get very far.

Manishearth 9y ago
http://www.cloudscan.me/2018/02/coretext-crash-osx-10133-lldb-voltron.html http://www.cloudscan.me/2018/02/coretext-crash-osx-10133-lld...

0x0 9y ago

Fonts are hard. I don't think CoreText is worse than others, most vendors seem to have a history of horrors, and at least it's not rendering fonts in a kernel driver like Windows did(!?) - see for example the Project Zero series of "one font vulnerability to rule them all" - https://googleprojectzero.blogspot.com/2015/07/one-font-vulnerability-to-rule-them-all.html https://googleprojectzero.blogspot.com/2015/07/one-font-vuln...

chris_wot 9y ago

Would fuzzing help with this?

bawana 9y ago

I went to the Apple Store in Burlington, MA yesterday (2/16/2018) None of the 'geniuses' there knew about this bug or how to fix it. They reminded me of microsoft drones following a troubleshooting script. 1) Reboot the phone 2) reset the phone 3) do a dfu restore.. After they did this my phone would not recognize its sim card. You see my phone originally had AT&T service. Then I switched to sprint some months back. I guess the dfu restore ended up relocking my phone to AT&T. At least that's what the Apple geniuses said. Then they said I had to go to a sprint store. This is the first time I was disappointed with Apple's human help. I remember past experiences going in there with a frayed power supply cord - they gave me a new one. On other occasions they replaced my phone when they couldn't fix it. They never spent more than 30 mins trying to fix a problem and always fixed it. Their knowledge was deep. This time their opinions were confabulatory - for example one guy said 'your phone does not have the antennae to work with sprint' EVEN THOUGH I reminded him I walked into the store with a working iPhone using sprint. Another one said that sprint jailbroke my phone and that's how they got it to work. WRONG. Another one said that the phone was 'soft unlocked' and relocked itself after the DFU restore. I felt sad for Apple because I know that's not what they were about. Then I suggested they replace my phone, the manager lady's eyes bugged out, her voice became stern and said ' There is no scenario where you walk out of here with a different phone'. This is the first time they wasted 5 hours of my time and actually caused me to have more trouble than what I walked in with. Galaxy here I come. or maybe pixel2