6 ms·

UTF-8 characters that behave oddly when the case is changed

72 points by Rendello 2y ago

Rendello 2y ago
I generated this list from a Python script I wrote a few months back for use in property tests in a Rust codebase. Its meant to break parsers that make bad assumptions about UTF-8, like assuming that upper- or lowercasing a character will always result in the character encoding having the same size in bytes, or even that it will result in one character. The "ﬀ" ligature, for example, is uppercased by Python as "FF", meaning it both becomes two separate characters, and is one byte smaller overall. I hope it's interesting.
- slome 2y ago
  Thanks for the insight. I had never considered this even though i researched quite some oddities in UTF-8 parsing myself over the years. It's the gift that keeps on giving when it comes to ways to breaking things in software, i find. Time to go over my code again.
- xg15 2y ago
  The implication of this is that there are also "roundtrip-unsafe" characters, i.e. flip_case(flip_case(x)) != x, right? Not sure I wanted to know...
  - TacticalCoder 2y ago
    > Not sure I wanted to know... Oh that's Unicode for you. It's not that they're "roundtrip unsafe", it's just that Unicode is a total and complete clusterfuck. Bruce Schneier in 2000 on Unicode security risks: https://www.schneier.com/crypto-gram/archives/2000/0715.html#9 https://www.schneier.com/crypto-gram/archives/2000/0715.html... Of course the attacks he envisioned materialized, like homoglyph attacks using internationalized domain names. My favorite line from Schneier: "Unicode is just too complex to ever be secure". And, no matter if you love Unicode or not, there's lots of wisdom in there. When design-by-committee gives birth to something way too complex, insecurity is never far behind.
    - moefh 2y ago
      > it's just that Unicode is a total and complete clusterfuck [...] > When design-by-committee gives birth to something way too complex, insecurity is never far behind. Human writing is (and has historically been) a "clusterfuck". Any system that's designed to encode every single known human writing system is bound to be way too complex. I almost always side with blaming systems that are too complex or insecure by design as opposed to blaming the users (the canonical example being C++), but in the case of Unicode there's no way to make a simpler system; we'll keep having problems until people stop treating Unicode text as something that works more or less like English or Western European text. In other words: if your code is doing input validation over an untrusted Unicode string in the year of our Lord 2024, no one is to blame but yourself. (That's not to say the Unicode committee didn't make some blunders along the way -- for instance the Han unification was heavily criticized -- but those have nothing to do with the problems described by Schneier).
    - SAI_Peregrinus 2y ago
      Unicode is worse than design-by-committee. It's a design-by-committee attempt to represent several hundred design-by-culture systems in one unified whole. Desgin-by-culture is even messier than design-by-committee, since everyone in the culture contributes to the design and there's never a formal specification, you just have to observe how something is used!
    - Sharlin 2y ago
      How could you ever make it simple given that the problem domain itself is complex as fuck? Should we all just have stuck with code pages and proprietary character encodings? Or just have people unable to use their own languages? Or even to spell their own names? It’s easy for a culturally blind English speaker to complain that text should be simple, must be due to design by committee that it isn’t!
    - Arnt 2y ago
      Could you try an argument that unicode is insecure compared to roll-your-own support for the necessary scripts? You may consider "necessary" to mean "the ones used in countries where at least two of Microsoft, Apple and Sun sold localised OSes".
  - LegionMammal978 2y ago
    A standard example here is the Turkish dotless I, which yields "ı" → "I" → "i" with most case-conversion libraries.
    - Macha 2y ago
      It feels like unifying it with the ASCII i is the mistake here. There should have just been 4 turkish characters in 2 pairs, rather than trying to reuse I/i It's not like we insist Α (Greek) is the same as A (Latin) or А (Cyrillic) just because they're visually identical.
      - WorldMaker 2y ago
        But even with separate characters, you aren't safe because the ASCII "unification" isn't just Unicode's fault to begin with, in some cases it is historic/cultural in its own ways: German ß has distinct upper and lower case forms, but also has a complicated history of sometimes, depending on locale, the upper case form is "SS" rather than the upper-case form of ß. In many of those same locales the lower-case form of "SS" is "ss", not ß. It doesn't even try to round-trip, and that's sort of intentional/cultural.
        atoav 2y ago
        Uppercase ẞ exists since 2017, so before that using SS as a replacement was the correct way of doing things. That is relatively recent wh3n it comes tonthat kind of change
      - Arnt 2y ago
        When do you think that first mistake happened? (Pick a year, then think about why it didn't happen in that year.)
        Macha 2y ago
        When Unicode was being specced out originally I guess. There was more interest in unifying characters at that stage (see also the far more controversial Han unification)
  - JadeNB 2y ago
    Indeed, the parent already gives one: flip_case(flip_case("ﬀ")) = "ff". (Since it's hard to tell with what I guess is default ligature formation, at least in my browser, the first is an 'ﬀ' ligature and the second is two 'f's.)
  - automatic6131 2y ago
    > flip_case(flip_case(x)) == x Falsehoods programmers believe about strings...
  - Rendello 2y ago
    I know Halloween was yesterday but let's discover this horror together with some terrifying Python[1]! Turns out, yep. For upper → lower → upper we have: Ω ω Ω İ i̇ İ K k K Å å Å ẞ ß SS ϴ θ Θ For lower → upper → lower there are a lot more. 1. https://gist.github.com/rendello/4d8266b7c52bf0e98eab2073b38829d9 https://gist.github.com/rendello/4d8266b7c52bf0e98eab2073b38...
    - xg15 2y ago
      This is really cool! Thanks a lot for the effort! But yeah, I got the idea from GP's "ff" example, but I'm kinda shocked there are so many.
    - gerikson 2y ago
      Lowercasing the symbols for Ohm, Kelvin or Ångström makes no sense. For the Greek alphabet cases, isn't there canonical forms for this kind of stuff?
      - Am4TIfIsER0ppos 2y ago
        Are those first two something other than Greek Omega and ascii K?
        gerikson 2y ago
        Yes, the Ohm, Kelvin and Ångström have their codepoints in Unicode.
        Am4TIfIsER0ppos 2y ago
        fuck I hate unicode they are not different symbols
- zahlman 2y ago
  Another assumption worth testing is that casing round-trips: >>> 'ẞ'.lower().upper() 'SS'
  - Rendello 2y ago
    I ended up playing with just that in another thread: https://news.ycombinator.com/item?id=42020476 https://news.ycombinator.com/item?id=42020476
zzo38computer 2y ago
I wrote another comment relating to case folding: https://news.ycombinator.com/item?id=41784627 https://news.ycombinator.com/item?id=41784627
throwaway173920 2y ago
In one of my work projects it was the Turkish İ that gave us trouble. In some case-insensitive text searching code, we matched the lowercase query against the lowercase text, and had to handle cases like that specially to avoid reporting the wrong matching span in the original text, since the lowercase string would have a different length than the uppercase string. This was one of my first real-world projects and opened my eyes a bit to the importance of specifications and standards.
- johannes1234321 2y ago
  In PHP the Turkish locale caused quite some trouble. In some situations a different locale was used for compiling and for runtime while handling "case-insensiteve" identifiers, fo that sometimes names with an "I" could not be found anymore.
- pavel_lishin 2y ago
  Can't mention the Turkish case situation without mentioning the actual murder that took place because of it: https://languagelog.ldc.upenn.edu/nll/?p=73 https://languagelog.ldc.upenn.edu/nll/?p=73
  - Filligree 2y ago
    The murder is a tragedy, of course, but I would hesitate to blame the cellphone. There’s overreactions, and then there’s… this.
    - oguz-ismail 2y ago
      [flagged]
      - hinkley 2y ago
        Except in this case the phone was the dress.
- Rendello 2y ago
rmrfchik 2y ago
It's not UTF-8 characters but Unicode.
- jonhohle 2y ago
  If you look at the list, it’s primarily (but not completely) about oddities in their UTF-8 encoding. Most of them appear to be on the boundary of adding additional bytes when the case is changed. That’s not really Unicode’s concern. There are also some that appear to change from single characters to grapheme clusters, which would be a Unicode quirk.
- Aardwolf 2y ago
  The byte-changes listed are for the UTF-8 encoding though, so it's about UTF-8 in that sense
- Retr0id 2y ago
  It's both.
  - zahlman 2y ago
    UTF-8 is simply an encoding; "UTF-8 characters" is just not correct use of language. Just like, say, "binary number"; a number has the same value regardless of the base you use to write it, and the base is a scheme for representing it, not a system for defining what a number is. This is a common imprecision in language which I have seen cause serious difficulties in learning concepts properly.
    - Retr0id 2y ago
      "unicode codepoint sequences whose codepoint lengths and/or utf8-code-unit-lengths behave oddly when you change their case" would not fit in a HN title, however
      - Rendello 2y ago
        I (OP) said above that "Unicode codepoints that expand or contract when case is changed in UTF-8" would have worked fine, I've changed the Gist title to that in any case. I'm curious if it would've affected the attention it received on HN.
D-Coder 2y ago
Raymond Chen's "Old New Thing" blog just commented on a similar issue: What has case distinction but is neither uppercase nor lowercase? https://devblogs.microsoft.com/oldnewthing/20241031-00/?p=110443 https://devblogs.microsoft.com/oldnewthing/20241031-00/?p=11...
LukeShu 2y ago
This doesn't include the oddest of all: sigma. When lowercasing Σ (U+03A3 Greek capital letter sigma) it is context-sensitive (on whether it is at the end of a word) whether it becomes σ (U+03C3 Greek small letter sigma) or ς (U+03C2 Greek small letter final sigma).
- BoxOfRain 2y ago
  That reminds me of the old 'long S'[1] that used to exist in English and survives in the symbol for integration. That worked in a ſimlar way for writing Engliſh, at the ſtart and middle of words you'd use the long s but not at the end so you end up with 'poſſeſs' for 'possess'. There were other rules around it too, I think you'd always use the usual S for a capital. [1]https://en.wikipedia.org/wiki/Long_s https://en.wikipedia.org/wiki/Long_s
  - hanche 2y ago
    Not only in English. My local newspaper (in Trondheim, Norway) shows its name as Adresſeaviſen on the front page (in a fractur font to boot).
- Rendello 2y ago
  True! This list could more accurately be described as "Unicode codepoints that expand or contract when case is changed in UTF-8", which is exactly what I was testing in my program. I had built a parser that was relying on some assumptions that I felt was not correct, so I built some tests with this data. For those interested, this was the generation script. I'm sure there was a way to do it better or simpler, and I wish I could just say this was a quick-and-dirty script, but in fact I spent quite a few hours on it (this is the fourth rewrite): https://gist.github.com/rendello/b06ca3d976d26fa011897bd1603ea044 https://gist.github.com/rendello/b06ca3d976d26fa011897bd1603...
- Rendello 2y ago
  Σ now shows up on my Unicode round-trip horror show ;) https://news.ycombinator.com/item?id=42020476 https://news.ycombinator.com/item?id=42020476
hgs3 2y ago
This isn't "odd" behavior. It's a consequence of using a multibyte encoding scheme. Also, when dealing with case mapping, you can't assume that the character count will remain constant. This is because in Unicode full case mappings can map a character to multiple characters, meaning you might end up with more characters than you started with, regardless of the encoding used.
- Rendello 2y ago
  That's exactly right. My comment here is related: https://news.ycombinator.com/item?id=42018937 https://news.ycombinator.com/item?id=42018937
layer8 2y ago
The canonical source data for this is https://www.unicode.org/Public/16.0.0/ucd/CaseFolding.txt https://www.unicode.org/Public/16.0.0/ucd/CaseFolding.txt, by the way.

72 points by Rendello 2y ago

I generated this list from a Python script I wrote a few months back for use in property tests in a Rust codebase. Its meant to break parsers that make bad assumptions about UTF-8, like assuming that upper- or lowercasing a character will always result in the character encoding having the same size in bytes, or even that it will result in one character. The "ﬀ" ligature, for example, is uppercased by Python as "FF", meaning it both becomes two separate characters, and is one byte smaller overall. I hope it's interesting.

slome 2y ago
Thanks for the insight. I had never considered this even though i researched quite some oddities in UTF-8 parsing myself over the years. It's the gift that keeps on giving when it comes to ways to breaking things in software, i find. Time to go over my code again.
xg15 2y ago
The implication of this is that there are also "roundtrip-unsafe" characters, i.e. flip_case(flip_case(x)) != x, right? Not sure I wanted to know...
- TacticalCoder 2y ago
  > Not sure I wanted to know... Oh that's Unicode for you. It's not that they're "roundtrip unsafe", it's just that Unicode is a total and complete clusterfuck. Bruce Schneier in 2000 on Unicode security risks: https://www.schneier.com/crypto-gram/archives/2000/0715.html#9 https://www.schneier.com/crypto-gram/archives/2000/0715.html... Of course the attacks he envisioned materialized, like homoglyph attacks using internationalized domain names. My favorite line from Schneier: "Unicode is just too complex to ever be secure". And, no matter if you love Unicode or not, there's lots of wisdom in there. When design-by-committee gives birth to something way too complex, insecurity is never far behind.
  - moefh 2y ago
    > it's just that Unicode is a total and complete clusterfuck [...] > When design-by-committee gives birth to something way too complex, insecurity is never far behind. Human writing is (and has historically been) a "clusterfuck". Any system that's designed to encode every single known human writing system is bound to be way too complex. I almost always side with blaming systems that are too complex or insecure by design as opposed to blaming the users (the canonical example being C++), but in the case of Unicode there's no way to make a simpler system; we'll keep having problems until people stop treating Unicode text as something that works more or less like English or Western European text. In other words: if your code is doing input validation over an untrusted Unicode string in the year of our Lord 2024, no one is to blame but yourself. (That's not to say the Unicode committee didn't make some blunders along the way -- for instance the Han unification was heavily criticized -- but those have nothing to do with the problems described by Schneier).
  - SAI_Peregrinus 2y ago
    Unicode is worse than design-by-committee. It's a design-by-committee attempt to represent several hundred design-by-culture systems in one unified whole. Desgin-by-culture is even messier than design-by-committee, since everyone in the culture contributes to the design and there's never a formal specification, you just have to observe how something is used!
  - Sharlin 2y ago
    How could you ever make it simple given that the problem domain itself is complex as fuck? Should we all just have stuck with code pages and proprietary character encodings? Or just have people unable to use their own languages? Or even to spell their own names? It’s easy for a culturally blind English speaker to complain that text should be simple, must be due to design by committee that it isn’t!
  - Arnt 2y ago
    Could you try an argument that unicode is insecure compared to roll-your-own support for the necessary scripts? You may consider "necessary" to mean "the ones used in countries where at least two of Microsoft, Apple and Sun sold localised OSes".
- LegionMammal978 2y ago
  A standard example here is the Turkish dotless I, which yields "ı" → "I" → "i" with most case-conversion libraries.
  - Macha 2y ago
    It feels like unifying it with the ASCII i is the mistake here. There should have just been 4 turkish characters in 2 pairs, rather than trying to reuse I/i It's not like we insist Α (Greek) is the same as A (Latin) or А (Cyrillic) just because they're visually identical.
    - WorldMaker 2y ago
      But even with separate characters, you aren't safe because the ASCII "unification" isn't just Unicode's fault to begin with, in some cases it is historic/cultural in its own ways: German ß has distinct upper and lower case forms, but also has a complicated history of sometimes, depending on locale, the upper case form is "SS" rather than the upper-case form of ß. In many of those same locales the lower-case form of "SS" is "ss", not ß. It doesn't even try to round-trip, and that's sort of intentional/cultural.
      - atoav 2y ago
        Uppercase ẞ exists since 2017, so before that using SS as a replacement was the correct way of doing things. That is relatively recent wh3n it comes tonthat kind of change
    - Arnt 2y ago
      When do you think that first mistake happened? (Pick a year, then think about why it didn't happen in that year.)
      - Macha 2y ago
        When Unicode was being specced out originally I guess. There was more interest in unifying characters at that stage (see also the far more controversial Han unification)
- JadeNB 2y ago
  Indeed, the parent already gives one: flip_case(flip_case("ﬀ")) = "ff". (Since it's hard to tell with what I guess is default ligature formation, at least in my browser, the first is an 'ﬀ' ligature and the second is two 'f's.)
- automatic6131 2y ago
  > flip_case(flip_case(x)) == x Falsehoods programmers believe about strings...
- Rendello 2y ago
  I know Halloween was yesterday but let's discover this horror together with some terrifying Python[1]! Turns out, yep. For upper → lower → upper we have: Ω ω Ω İ i̇ İ K k K Å å Å ẞ ß SS ϴ θ Θ For lower → upper → lower there are a lot more. 1. https://gist.github.com/rendello/4d8266b7c52bf0e98eab2073b38829d9 https://gist.github.com/rendello/4d8266b7c52bf0e98eab2073b38...
  - xg15 2y ago
    This is really cool! Thanks a lot for the effort! But yeah, I got the idea from GP's "ff" example, but I'm kinda shocked there are so many.
  - gerikson 2y ago
    Lowercasing the symbols for Ohm, Kelvin or Ångström makes no sense. For the Greek alphabet cases, isn't there canonical forms for this kind of stuff?
    - Am4TIfIsER0ppos 2y ago
      Are those first two something other than Greek Omega and ascii K?
      - gerikson 2y ago
        Yes, the Ohm, Kelvin and Ångström have their codepoints in Unicode.
        Am4TIfIsER0ppos 2y ago
        fuck I hate unicode they are not different symbols
zahlman 2y ago
Another assumption worth testing is that casing round-trips: >>> 'ẞ'.lower().upper() 'SS'
- Rendello 2y ago
  I ended up playing with just that in another thread: https://news.ycombinator.com/item?id=42020476 https://news.ycombinator.com/item?id=42020476

zzo38computer 2y ago

I wrote another comment relating to case folding: https://news.ycombinator.com/item?id=41784627 https://news.ycombinator.com/item?id=41784627

throwaway173920 2y ago

In one of my work projects it was the Turkish İ that gave us trouble. In some case-insensitive text searching code, we matched the lowercase query against the lowercase text, and had to handle cases like that specially to avoid reporting the wrong matching span in the original text, since the lowercase string would have a different length than the uppercase string. This was one of my first real-world projects and opened my eyes a bit to the importance of specifications and standards.

johannes1234321 2y ago
In PHP the Turkish locale caused quite some trouble. In some situations a different locale was used for compiling and for runtime while handling "case-insensiteve" identifiers, fo that sometimes names with an "I" could not be found anymore.
pavel_lishin 2y ago
Can't mention the Turkish case situation without mentioning the actual murder that took place because of it: https://languagelog.ldc.upenn.edu/nll/?p=73 https://languagelog.ldc.upenn.edu/nll/?p=73
- Filligree 2y ago
  The murder is a tragedy, of course, but I would hesitate to blame the cellphone. There’s overreactions, and then there’s… this.
  - oguz-ismail 2y ago
    [flagged]
    - hinkley 2y ago
      Except in this case the phone was the dress.
Rendello 2y ago

rmrfchik 2y ago

It's not UTF-8 characters but Unicode.

jonhohle 2y ago
If you look at the list, it’s primarily (but not completely) about oddities in their UTF-8 encoding. Most of them appear to be on the boundary of adding additional bytes when the case is changed. That’s not really Unicode’s concern. There are also some that appear to change from single characters to grapheme clusters, which would be a Unicode quirk.
Aardwolf 2y ago
The byte-changes listed are for the UTF-8 encoding though, so it's about UTF-8 in that sense
Retr0id 2y ago
It's both.
- zahlman 2y ago
  UTF-8 is simply an encoding; "UTF-8 characters" is just not correct use of language. Just like, say, "binary number"; a number has the same value regardless of the base you use to write it, and the base is a scheme for representing it, not a system for defining what a number is. This is a common imprecision in language which I have seen cause serious difficulties in learning concepts properly.
  - Retr0id 2y ago
    "unicode codepoint sequences whose codepoint lengths and/or utf8-code-unit-lengths behave oddly when you change their case" would not fit in a HN title, however
    - Rendello 2y ago
      I (OP) said above that "Unicode codepoints that expand or contract when case is changed in UTF-8" would have worked fine, I've changed the Gist title to that in any case. I'm curious if it would've affected the attention it received on HN.

D-Coder 2y ago

Raymond Chen's "Old New Thing" blog just commented on a similar issue: What has case distinction but is neither uppercase nor lowercase? https://devblogs.microsoft.com/oldnewthing/20241031-00/?p=110443 https://devblogs.microsoft.com/oldnewthing/20241031-00/?p=11...

LukeShu 2y ago

This doesn't include the oddest of all: sigma. When lowercasing Σ (U+03A3 Greek capital letter sigma) it is context-sensitive (on whether it is at the end of a word) whether it becomes σ (U+03C3 Greek small letter sigma) or ς (U+03C2 Greek small letter final sigma).

BoxOfRain 2y ago
That reminds me of the old 'long S'[1] that used to exist in English and survives in the symbol for integration. That worked in a ſimlar way for writing Engliſh, at the ſtart and middle of words you'd use the long s but not at the end so you end up with 'poſſeſs' for 'possess'. There were other rules around it too, I think you'd always use the usual S for a capital. [1]https://en.wikipedia.org/wiki/Long_s https://en.wikipedia.org/wiki/Long_s
- hanche 2y ago
  Not only in English. My local newspaper (in Trondheim, Norway) shows its name as Adresſeaviſen on the front page (in a fractur font to boot).
Rendello 2y ago
True! This list could more accurately be described as "Unicode codepoints that expand or contract when case is changed in UTF-8", which is exactly what I was testing in my program. I had built a parser that was relying on some assumptions that I felt was not correct, so I built some tests with this data. For those interested, this was the generation script. I'm sure there was a way to do it better or simpler, and I wish I could just say this was a quick-and-dirty script, but in fact I spent quite a few hours on it (this is the fourth rewrite): https://gist.github.com/rendello/b06ca3d976d26fa011897bd1603ea044 https://gist.github.com/rendello/b06ca3d976d26fa011897bd1603...
Rendello 2y ago
Σ now shows up on my Unicode round-trip horror show ;) https://news.ycombinator.com/item?id=42020476 https://news.ycombinator.com/item?id=42020476

hgs3 2y ago

This isn't "odd" behavior. It's a consequence of using a multibyte encoding scheme. Also, when dealing with case mapping, you can't assume that the character count will remain constant. This is because in Unicode full case mappings can map a character to multiple characters, meaning you might end up with more characters than you started with, regardless of the encoding used.

Rendello 2y ago
That's exactly right. My comment here is related: https://news.ycombinator.com/item?id=42018937 https://news.ycombinator.com/item?id=42018937

layer8 2y ago

The canonical source data for this is https://www.unicode.org/Public/16.0.0/ucd/CaseFolding.txt https://www.unicode.org/Public/16.0.0/ucd/CaseFolding.txt, by the way.