5 ms·

I generated this list from a Python script I wrote a few months back for use in property tests in a Rust codebase. Its meant to break parsers that make bad assu

I generated this list from a Python script I wrote a few months back for use in property tests in a Rust codebase. Its meant to break parsers that make bad assumptions about UTF-8, like assuming that upper- or lowercasing a character will always result in the character encoding having the same size in bytes, or even that it will result in one character. The "ﬀ" ligature, for example, is uppercased by Python as "FF", meaning it both becomes two separate characters, and is one byte smaller overall. I hope it's interesting.

slome 2y ago
Thanks for the insight. I had never considered this even though i researched quite some oddities in UTF-8 parsing myself over the years. It's the gift that keeps on giving when it comes to ways to breaking things in software, i find. Time to go over my code again.
xg15 2y ago
The implication of this is that there are also "roundtrip-unsafe" characters, i.e. flip_case(flip_case(x)) != x, right? Not sure I wanted to know...
- TacticalCoder 2y ago
  > Not sure I wanted to know... Oh that's Unicode for you. It's not that they're "roundtrip unsafe", it's just that Unicode is a total and complete clusterfuck. Bruce Schneier in 2000 on Unicode security risks: https://www.schneier.com/crypto-gram/archives/2000/0715.html#9 https://www.schneier.com/crypto-gram/archives/2000/0715.html... Of course the attacks he envisioned materialized, like homoglyph attacks using internationalized domain names. My favorite line from Schneier: "Unicode is just too complex to ever be secure". And, no matter if you love Unicode or not, there's lots of wisdom in there. When design-by-committee gives birth to something way too complex, insecurity is never far behind.
  - moefh 2y ago
    > it's just that Unicode is a total and complete clusterfuck [...] > When design-by-committee gives birth to something way too complex, insecurity is never far behind. Human writing is (and has historically been) a "clusterfuck". Any system that's designed to encode every single known human writing system is bound to be way too complex. I almost always side with blaming systems that are too complex or insecure by design as opposed to blaming the users (the canonical example being C++), but in the case of Unicode there's no way to make a simpler system; we'll keep having problems until people stop treating Unicode text as something that works more or less like English or Western European text. In other words: if your code is doing input validation over an untrusted Unicode string in the year of our Lord 2024, no one is to blame but yourself. (That's not to say the Unicode committee didn't make some blunders along the way -- for instance the Han unification was heavily criticized -- but those have nothing to do with the problems described by Schneier).
  - SAI_Peregrinus 2y ago
    Unicode is worse than design-by-committee. It's a design-by-committee attempt to represent several hundred design-by-culture systems in one unified whole. Desgin-by-culture is even messier than design-by-committee, since everyone in the culture contributes to the design and there's never a formal specification, you just have to observe how something is used!
  - Sharlin 2y ago
    How could you ever make it simple given that the problem domain itself is complex as fuck? Should we all just have stuck with code pages and proprietary character encodings? Or just have people unable to use their own languages? Or even to spell their own names? It’s easy for a culturally blind English speaker to complain that text should be simple, must be due to design by committee that it isn’t!
  - Arnt 2y ago
    Could you try an argument that unicode is insecure compared to roll-your-own support for the necessary scripts? You may consider "necessary" to mean "the ones used in countries where at least two of Microsoft, Apple and Sun sold localised OSes".
- LegionMammal978 2y ago
  A standard example here is the Turkish dotless I, which yields "ı" → "I" → "i" with most case-conversion libraries.
  - Macha 2y ago
    It feels like unifying it with the ASCII i is the mistake here. There should have just been 4 turkish characters in 2 pairs, rather than trying to reuse I/i It's not like we insist Α (Greek) is the same as A (Latin) or А (Cyrillic) just because they're visually identical.
    - WorldMaker 2y ago
      But even with separate characters, you aren't safe because the ASCII "unification" isn't just Unicode's fault to begin with, in some cases it is historic/cultural in its own ways: German ß has distinct upper and lower case forms, but also has a complicated history of sometimes, depending on locale, the upper case form is "SS" rather than the upper-case form of ß. In many of those same locales the lower-case form of "SS" is "ss", not ß. It doesn't even try to round-trip, and that's sort of intentional/cultural.
      - atoav 2y ago
        Uppercase ẞ exists since 2017, so before that using SS as a replacement was the correct way of doing things. That is relatively recent wh3n it comes tonthat kind of change
    - Arnt 2y ago
      When do you think that first mistake happened? (Pick a year, then think about why it didn't happen in that year.)
      - Macha 2y ago
        When Unicode was being specced out originally I guess. There was more interest in unifying characters at that stage (see also the far more controversial Han unification)
- JadeNB 2y ago
  Indeed, the parent already gives one: flip_case(flip_case("ﬀ")) = "ff". (Since it's hard to tell with what I guess is default ligature formation, at least in my browser, the first is an 'ﬀ' ligature and the second is two 'f's.)
- automatic6131 2y ago
  > flip_case(flip_case(x)) == x Falsehoods programmers believe about strings...
- Rendello 2y ago
  I know Halloween was yesterday but let's discover this horror together with some terrifying Python[1]! Turns out, yep. For upper → lower → upper we have: Ω ω Ω İ i̇ İ K k K Å å Å ẞ ß SS ϴ θ Θ For lower → upper → lower there are a lot more. 1. https://gist.github.com/rendello/4d8266b7c52bf0e98eab2073b38829d9 https://gist.github.com/rendello/4d8266b7c52bf0e98eab2073b38...
  - xg15 2y ago
    This is really cool! Thanks a lot for the effort! But yeah, I got the idea from GP's "ff" example, but I'm kinda shocked there are so many.
  - gerikson 2y ago
    Lowercasing the symbols for Ohm, Kelvin or Ångström makes no sense. For the Greek alphabet cases, isn't there canonical forms for this kind of stuff?
    - Am4TIfIsER0ppos 2y ago
      Are those first two something other than Greek Omega and ascii K?
      - gerikson 2y ago
        Yes, the Ohm, Kelvin and Ångström have their codepoints in Unicode.
        Am4TIfIsER0ppos 2y ago
        fuck I hate unicode they are not different symbols
zahlman 2y ago
Another assumption worth testing is that casing round-trips: >>> 'ẞ'.lower().upper() 'SS'
- Rendello 2y ago
  I ended up playing with just that in another thread: https://news.ycombinator.com/item?id=42020476 https://news.ycombinator.com/item?id=42020476

slome 2y ago

Thanks for the insight. I had never considered this even though i researched quite some oddities in UTF-8 parsing myself over the years. It's the gift that keeps on giving when it comes to ways to breaking things in software, i find. Time to go over my code again.

xg15 2y ago

The implication of this is that there are also "roundtrip-unsafe" characters, i.e. flip_case(flip_case(x)) != x, right? Not sure I wanted to know...

TacticalCoder 2y ago
> Not sure I wanted to know... Oh that's Unicode for you. It's not that they're "roundtrip unsafe", it's just that Unicode is a total and complete clusterfuck. Bruce Schneier in 2000 on Unicode security risks: https://www.schneier.com/crypto-gram/archives/2000/0715.html#9 https://www.schneier.com/crypto-gram/archives/2000/0715.html... Of course the attacks he envisioned materialized, like homoglyph attacks using internationalized domain names. My favorite line from Schneier: "Unicode is just too complex to ever be secure". And, no matter if you love Unicode or not, there's lots of wisdom in there. When design-by-committee gives birth to something way too complex, insecurity is never far behind.
- moefh 2y ago
  > it's just that Unicode is a total and complete clusterfuck [...] > When design-by-committee gives birth to something way too complex, insecurity is never far behind. Human writing is (and has historically been) a "clusterfuck". Any system that's designed to encode every single known human writing system is bound to be way too complex. I almost always side with blaming systems that are too complex or insecure by design as opposed to blaming the users (the canonical example being C++), but in the case of Unicode there's no way to make a simpler system; we'll keep having problems until people stop treating Unicode text as something that works more or less like English or Western European text. In other words: if your code is doing input validation over an untrusted Unicode string in the year of our Lord 2024, no one is to blame but yourself. (That's not to say the Unicode committee didn't make some blunders along the way -- for instance the Han unification was heavily criticized -- but those have nothing to do with the problems described by Schneier).
- SAI_Peregrinus 2y ago
  Unicode is worse than design-by-committee. It's a design-by-committee attempt to represent several hundred design-by-culture systems in one unified whole. Desgin-by-culture is even messier than design-by-committee, since everyone in the culture contributes to the design and there's never a formal specification, you just have to observe how something is used!
- Sharlin 2y ago
  How could you ever make it simple given that the problem domain itself is complex as fuck? Should we all just have stuck with code pages and proprietary character encodings? Or just have people unable to use their own languages? Or even to spell their own names? It’s easy for a culturally blind English speaker to complain that text should be simple, must be due to design by committee that it isn’t!
- Arnt 2y ago
  Could you try an argument that unicode is insecure compared to roll-your-own support for the necessary scripts? You may consider "necessary" to mean "the ones used in countries where at least two of Microsoft, Apple and Sun sold localised OSes".
LegionMammal978 2y ago
A standard example here is the Turkish dotless I, which yields "ı" → "I" → "i" with most case-conversion libraries.
- Macha 2y ago
  It feels like unifying it with the ASCII i is the mistake here. There should have just been 4 turkish characters in 2 pairs, rather than trying to reuse I/i It's not like we insist Α (Greek) is the same as A (Latin) or А (Cyrillic) just because they're visually identical.
  - WorldMaker 2y ago
    But even with separate characters, you aren't safe because the ASCII "unification" isn't just Unicode's fault to begin with, in some cases it is historic/cultural in its own ways: German ß has distinct upper and lower case forms, but also has a complicated history of sometimes, depending on locale, the upper case form is "SS" rather than the upper-case form of ß. In many of those same locales the lower-case form of "SS" is "ss", not ß. It doesn't even try to round-trip, and that's sort of intentional/cultural.
    - atoav 2y ago
      Uppercase ẞ exists since 2017, so before that using SS as a replacement was the correct way of doing things. That is relatively recent wh3n it comes tonthat kind of change
  - Arnt 2y ago
    When do you think that first mistake happened? (Pick a year, then think about why it didn't happen in that year.)
    - Macha 2y ago
      When Unicode was being specced out originally I guess. There was more interest in unifying characters at that stage (see also the far more controversial Han unification)
JadeNB 2y ago
Indeed, the parent already gives one: flip_case(flip_case("ﬀ")) = "ff". (Since it's hard to tell with what I guess is default ligature formation, at least in my browser, the first is an 'ﬀ' ligature and the second is two 'f's.)
automatic6131 2y ago
> flip_case(flip_case(x)) == x Falsehoods programmers believe about strings...
Rendello 2y ago
I know Halloween was yesterday but let's discover this horror together with some terrifying Python[1]! Turns out, yep. For upper → lower → upper we have: Ω ω Ω İ i̇ İ K k K Å å Å ẞ ß SS ϴ θ Θ For lower → upper → lower there are a lot more. 1. https://gist.github.com/rendello/4d8266b7c52bf0e98eab2073b38829d9 https://gist.github.com/rendello/4d8266b7c52bf0e98eab2073b38...
- xg15 2y ago
  This is really cool! Thanks a lot for the effort! But yeah, I got the idea from GP's "ff" example, but I'm kinda shocked there are so many.
- gerikson 2y ago
  Lowercasing the symbols for Ohm, Kelvin or Ångström makes no sense. For the Greek alphabet cases, isn't there canonical forms for this kind of stuff?
  - Am4TIfIsER0ppos 2y ago
    Are those first two something other than Greek Omega and ascii K?
    - gerikson 2y ago
      Yes, the Ohm, Kelvin and Ångström have their codepoints in Unicode.
      - Am4TIfIsER0ppos 2y ago
        fuck I hate unicode they are not different symbols

zahlman 2y ago

Another assumption worth testing is that casing round-trips: >>> 'ẞ'.lower().upper() 'SS'

Rendello 2y ago
I ended up playing with just that in another thread: https://news.ycombinator.com/item?id=42020476 https://news.ycombinator.com/item?id=42020476