4 ms·

Unicode Footguns in Python

43 points by meander_water 1y ago

naIak 1y ago
I’m going to trigger some ptsd with this… UnicodeDecodeError
morshu9001 1y ago
Unicode footguns, in Python
OkayPhysicist 1y ago
Frankly, the key takeaway to most problems people run into with Unicode is that there are very, very few operations that are universally well-defined for arbitrary user-provided text. Pretty much the moment you step outside the realm of "receive, copy, save, regurgitate", you're probably going to run into edge cases.
dhosek 1y ago
Grapheme count is not a useful number. Even in a monospaced font, you’ll find that the grapheme count doesn’t give you a measurement of width since emoji will usually not be the same width as other characters.
- paulddraper 1y ago
  Grapheme count (or rather, indexing) is necessary to do text selection or cursor positions. Fortunately you can usually outsource this to a UI toolkit which can do it.
  - WorldMaker 1y ago
    Most of the UI toolkits defer to font-based layout calculation engines rather than grapheme counts. Grapheme counts are a handy approximation in many cases, but you aren't going to get truly accurate text selection or cursor positions without real geometry from a font and whatever layout engine is closest to it. (Fonts may disagree on supported ligatures, for instance, or not support an emoji grapheme cluster and fall back to displaying multiple component emoji, or layout multiple graphemes in two dimensions for a given language [CJK, Arabic, even Math symbols, even before factoring in if a font layout engine supports the optional but cool UnicodeMath [0]] or any number of other tweaks and distinctions between encoding a block of text and displaying a block of text.) [0] https://www.unicode.org/notes/tn28/UTN28-PlainTextMath-v3.1.pdf https://www.unicode.org/notes/tn28/UTN28-PlainTextMath-v3.1....
    - dhosek 1y ago
      Ligatures are a whole other thing independent of graphemes. Grapheme clustering is based entirely independently of the font and a sequence like U+1F1E6 U+1F1E6 is considered a single grapheme according to the Unicode specification even though no font will display this as a single character (it will instead be rendered as something resembling two A’s in boxes in most fonts).
      - WorldMaker 1y ago
        I included ligatures as a logically separate concept that needs to be accounted for when counting the "displayed width" of a string and how you handle cursor position and selection logic. However, ligatures are a part of grapheme clustering and it isn't entirely independent: Examples above and elsewhere include ligatures that have Unicode encodings (ex: ﬁ). Ligatures have been a part of character encodings and have affected grapheme clustering since EBCDIC (directly from which Unicode inherits a lot of its single codepoint Latin ligatures). There are way too many debates if normal forms should encode ligatures or always decompose them or some other strategy. The normal forms (NFC and NFD, NFKC and NFKD) themselves are embedded as a step of several of the standard grapheme clustering algorithms. Some people think ligatures should be entirely left to fonts and Unicode ligatures a relic of the IBM bitmap font past. Some fonts think it would be nice if some ligatures were more directly encoded and normal forms could do a first pass for them. Unicode has had portions of the specification on both sides of that argument. It generally leans away from the backward compatibility ligatures and generally normalizes them to decomposition, especially in locales like "en-us" these days, but it doesn't always and in every locale. (All of that is before you start to consider languages that are almost nothing but long sequences of ligatures, including but not limited to Arabic.) You can't do grapheme clustering and entirely ignore ligatures. You certainly can't count "display width" or "display position" without paying attention to ligatures, which was the point of bringing them up alongside mentions of grapheme clustering length and why it is insufficient for "display position" and "display width".
        dhosek 1y ago
        Umm, you’re confusing things here. When you said “several of the standard grapheme clustering algorithms” that indicated the first element of confusion. There is one grapheme clustering algorithm in Unicode.¹ Ligature combinations, whether encoded in Unicode or not are independent of the concept of graphemes and are very much font-dependent, as is whether or not a grapheme cluster is displayed as multiple characters or a single character (any two consecutive regional indicators are considered a single grapheme, whether they map to a national flag or not and the set of national flags and whether those flags are even displayed or the output is just the regional indicator symbol is up to the font). Normalization does not play a role in grapheme clustering at all, which is why á and a+acute accent are both considered single graphemes. I’m puzzled about the assertion of EBCDIC having Latin ligatures because I never encountered them in my IBM mainframe days and a casual search didn’t turn any up. The only 8-bit encoding that I was aware of that included any ligatures was Mac extended ASCII which included fi and fl (well, putting aside TeX’s 7-bit encoding which was only used by TeX and because of its using x20 to hold the acute accent was unusable by other applications which expected a blank character in that spot). The question about dealing with ligatures for non-Latin scripts generally came down to compatibility with existing systems more than anything else. This is why, for example, the Devanagari and Thai encodings which both have vowel markings which can occur before, after or around a consonant handle the sequence of input differently. Assembly of jambo into characters in Hangul is another case where theoretically, one could have all the displayed characters handled through ligatures and not encode the syllables directly (which is, in fact, how most modern Korean is encoded as evidenced by Korean wikipedia), but because the original Korean font encoding has all the syllables encoded in it² those syllables are part of Unicode as well. But the bottom line here, is that you seem to be confusing terminology a lot here. You can very much do grapheme clustering without paying attention to ligatures, and rules for things like normalized composition/decomposition are entirely independent of grapheme clustering (the K-forms of both, manage transitions like ¹ to 1 or ﬁ to fi and represent a one way transition). ⸻ 1. I wrote a Rust library implementing this and I’ve been following Unicode since it was a proposal from Microsoft and Apple in the 90s, so I know a little about this. 2. I think the more accurate term would be “most” of the syllables as I’ve seen additional syllables added to Unicode over time.
- Spivak 1y ago
  For certain use-cases, but it's not like any of the other usual notions of text length are any better for what you want.
  - lmm 1y ago
    If all possible notions of length are footguns, maybe there should be no default "length" operation available.
    - WorldMaker 1y ago
      Indeed. Several languages have debated dropping or already dropped an easy to access "Length" count on strings and making it much more explicit if you want "UTF-8 Encoded Length" or "Codepoint count" or "Grapheme count" or "Grapheme cluster count" or "Laid out font width". Why endorse a bad winner when you can make more of the trade-offs more obvious and give programmers a better chance of asking for the right information instead of using the wrong information because it is the default and assuming it is correct?
renhanxue 1y ago
The article has good tips, but Unicode normalization is just the tip of the iceberg. It is almost always impossible to do what your users expect without locale information (different languages and locales sort and compare the same graphemes differently). "What do we mean when we say two strings are equal" can be a surprisingly difficult question to answer. It's practical too, not philosophical. By the way, try looking up the standardized Unicode casefolding algorithm sometime, it is a thing to behold.
- Groxx 1y ago
  the normalization doc is interesting too imo: https://unicode.org/reports/tr15/ https://unicode.org/reports/tr15/ in particular, the differences between NFC and NFKC are "fun", and rather meaningful in many cases. e.g. NFC says that "fi" and "ﬁ" are different and not equal, though the latter is just a ligature of the former and is literally identical in meaning. this applies to ﬃ too. half vs full width Chinese characters are also "different" under NFC. NFKC makes those examples equal though... at the cost of saying "2⁵" is equal to "25". language is fun!
o11c 1y ago
I've said this before and have said it again: Python3 got rid of the wrong string type. With `bytes` it was obvious that byte length was not the same as $whatever length, and that was really the only semi-common bug (and was mostly limited to English speakers who are new to programming). All other bugs come from blindly trusting `unicode` whose bugs are far more subtle and numerous.
- Flimm 1y ago
  I strongly disagree. Python 2 had no bytes type to get rid of. It had a string type that could not handle code points above U+00FF at all, and could not handle code points above U+007F very well. In addition, Python 2 had a Unicode type, and the types would get automatically converted to each other and/or encoded/decoded, often incorrectly, and sometimes throwing runtime exceptions. Python 3 introduced the bytes type that you like so much. It sounds like you would enjoy a Python 4 with only a bytes type and no string type, and presumably with a strong convention to only use UTF-8 or with required encoding arguments everywhere. In both Python 2 and Python 3, you still have to learn how to handle grapheme clusters carefully.
- seanhunter 1y ago
  Python 3 didn't get rid of bytes though. If you want to manipulate data as bytes you absolutely can do that. https://docs.python.org/3/library/stdtypes.html#binary-sequence-types-bytes-bytearray-memoryview https://docs.python.org/3/library/stdtypes.html#binary-seque... "The core built-in types for manipulating binary data are bytes and bytearray."
  - o11c 1y ago
    Those are arrays of integers, not of bytes. Most bytes are character-ish, which only python2's bytes acknowledged. Additionally, python2 supported a much richer set of operations on its bytes type than python3 does.
deleted 1y ago
[deleted]

43 points by meander_water 1y ago

naIak 1y ago

I’m going to trigger some ptsd with this… UnicodeDecodeError

morshu9001 1y ago

Unicode footguns, in Python

OkayPhysicist 1y ago

Frankly, the key takeaway to most problems people run into with Unicode is that there are very, very few operations that are universally well-defined for arbitrary user-provided text. Pretty much the moment you step outside the realm of "receive, copy, save, regurgitate", you're probably going to run into edge cases.

dhosek 1y ago

Grapheme count is not a useful number. Even in a monospaced font, you’ll find that the grapheme count doesn’t give you a measurement of width since emoji will usually not be the same width as other characters.

paulddraper 1y ago
Grapheme count (or rather, indexing) is necessary to do text selection or cursor positions. Fortunately you can usually outsource this to a UI toolkit which can do it.
- WorldMaker 1y ago
  Most of the UI toolkits defer to font-based layout calculation engines rather than grapheme counts. Grapheme counts are a handy approximation in many cases, but you aren't going to get truly accurate text selection or cursor positions without real geometry from a font and whatever layout engine is closest to it. (Fonts may disagree on supported ligatures, for instance, or not support an emoji grapheme cluster and fall back to displaying multiple component emoji, or layout multiple graphemes in two dimensions for a given language [CJK, Arabic, even Math symbols, even before factoring in if a font layout engine supports the optional but cool UnicodeMath [0]] or any number of other tweaks and distinctions between encoding a block of text and displaying a block of text.) [0] https://www.unicode.org/notes/tn28/UTN28-PlainTextMath-v3.1.pdf https://www.unicode.org/notes/tn28/UTN28-PlainTextMath-v3.1....
  - dhosek 1y ago
    Ligatures are a whole other thing independent of graphemes. Grapheme clustering is based entirely independently of the font and a sequence like U+1F1E6 U+1F1E6 is considered a single grapheme according to the Unicode specification even though no font will display this as a single character (it will instead be rendered as something resembling two A’s in boxes in most fonts).
    - WorldMaker 1y ago
      I included ligatures as a logically separate concept that needs to be accounted for when counting the "displayed width" of a string and how you handle cursor position and selection logic. However, ligatures are a part of grapheme clustering and it isn't entirely independent: Examples above and elsewhere include ligatures that have Unicode encodings (ex: ﬁ). Ligatures have been a part of character encodings and have affected grapheme clustering since EBCDIC (directly from which Unicode inherits a lot of its single codepoint Latin ligatures). There are way too many debates if normal forms should encode ligatures or always decompose them or some other strategy. The normal forms (NFC and NFD, NFKC and NFKD) themselves are embedded as a step of several of the standard grapheme clustering algorithms. Some people think ligatures should be entirely left to fonts and Unicode ligatures a relic of the IBM bitmap font past. Some fonts think it would be nice if some ligatures were more directly encoded and normal forms could do a first pass for them. Unicode has had portions of the specification on both sides of that argument. It generally leans away from the backward compatibility ligatures and generally normalizes them to decomposition, especially in locales like "en-us" these days, but it doesn't always and in every locale. (All of that is before you start to consider languages that are almost nothing but long sequences of ligatures, including but not limited to Arabic.) You can't do grapheme clustering and entirely ignore ligatures. You certainly can't count "display width" or "display position" without paying attention to ligatures, which was the point of bringing them up alongside mentions of grapheme clustering length and why it is insufficient for "display position" and "display width".
      - dhosek 1y ago
        Umm, you’re confusing things here. When you said “several of the standard grapheme clustering algorithms” that indicated the first element of confusion. There is one grapheme clustering algorithm in Unicode.¹ Ligature combinations, whether encoded in Unicode or not are independent of the concept of graphemes and are very much font-dependent, as is whether or not a grapheme cluster is displayed as multiple characters or a single character (any two consecutive regional indicators are considered a single grapheme, whether they map to a national flag or not and the set of national flags and whether those flags are even displayed or the output is just the regional indicator symbol is up to the font). Normalization does not play a role in grapheme clustering at all, which is why á and a+acute accent are both considered single graphemes. I’m puzzled about the assertion of EBCDIC having Latin ligatures because I never encountered them in my IBM mainframe days and a casual search didn’t turn any up. The only 8-bit encoding that I was aware of that included any ligatures was Mac extended ASCII which included fi and fl (well, putting aside TeX’s 7-bit encoding which was only used by TeX and because of its using x20 to hold the acute accent was unusable by other applications which expected a blank character in that spot). The question about dealing with ligatures for non-Latin scripts generally came down to compatibility with existing systems more than anything else. This is why, for example, the Devanagari and Thai encodings which both have vowel markings which can occur before, after or around a consonant handle the sequence of input differently. Assembly of jambo into characters in Hangul is another case where theoretically, one could have all the displayed characters handled through ligatures and not encode the syllables directly (which is, in fact, how most modern Korean is encoded as evidenced by Korean wikipedia), but because the original Korean font encoding has all the syllables encoded in it² those syllables are part of Unicode as well. But the bottom line here, is that you seem to be confusing terminology a lot here. You can very much do grapheme clustering without paying attention to ligatures, and rules for things like normalized composition/decomposition are entirely independent of grapheme clustering (the K-forms of both, manage transitions like ¹ to 1 or ﬁ to fi and represent a one way transition). ⸻ 1. I wrote a Rust library implementing this and I’ve been following Unicode since it was a proposal from Microsoft and Apple in the 90s, so I know a little about this. 2. I think the more accurate term would be “most” of the syllables as I’ve seen additional syllables added to Unicode over time.
Spivak 1y ago
For certain use-cases, but it's not like any of the other usual notions of text length are any better for what you want.
- lmm 1y ago
  If all possible notions of length are footguns, maybe there should be no default "length" operation available.
  - WorldMaker 1y ago
    Indeed. Several languages have debated dropping or already dropped an easy to access "Length" count on strings and making it much more explicit if you want "UTF-8 Encoded Length" or "Codepoint count" or "Grapheme count" or "Grapheme cluster count" or "Laid out font width". Why endorse a bad winner when you can make more of the trade-offs more obvious and give programmers a better chance of asking for the right information instead of using the wrong information because it is the default and assuming it is correct?

renhanxue 1y ago

The article has good tips, but Unicode normalization is just the tip of the iceberg. It is almost always impossible to do what your users expect without locale information (different languages and locales sort and compare the same graphemes differently). "What do we mean when we say two strings are equal" can be a surprisingly difficult question to answer. It's practical too, not philosophical. By the way, try looking up the standardized Unicode casefolding algorithm sometime, it is a thing to behold.

Groxx 1y ago
the normalization doc is interesting too imo: https://unicode.org/reports/tr15/ https://unicode.org/reports/tr15/ in particular, the differences between NFC and NFKC are "fun", and rather meaningful in many cases. e.g. NFC says that "fi" and "ﬁ" are different and not equal, though the latter is just a ligature of the former and is literally identical in meaning. this applies to ﬃ too. half vs full width Chinese characters are also "different" under NFC. NFKC makes those examples equal though... at the cost of saying "2⁵" is equal to "25". language is fun!

o11c 1y ago

I've said this before and have said it again: Python3 got rid of the wrong string type. With `bytes` it was obvious that byte length was not the same as $whatever length, and that was really the only semi-common bug (and was mostly limited to English speakers who are new to programming). All other bugs come from blindly trusting `unicode` whose bugs are far more subtle and numerous.

Flimm 1y ago
I strongly disagree. Python 2 had no bytes type to get rid of. It had a string type that could not handle code points above U+00FF at all, and could not handle code points above U+007F very well. In addition, Python 2 had a Unicode type, and the types would get automatically converted to each other and/or encoded/decoded, often incorrectly, and sometimes throwing runtime exceptions. Python 3 introduced the bytes type that you like so much. It sounds like you would enjoy a Python 4 with only a bytes type and no string type, and presumably with a strong convention to only use UTF-8 or with required encoding arguments everywhere. In both Python 2 and Python 3, you still have to learn how to handle grapheme clusters carefully.
seanhunter 1y ago
Python 3 didn't get rid of bytes though. If you want to manipulate data as bytes you absolutely can do that. https://docs.python.org/3/library/stdtypes.html#binary-sequence-types-bytes-bytearray-memoryview https://docs.python.org/3/library/stdtypes.html#binary-seque... "The core built-in types for manipulating binary data are bytes and bytearray."
- o11c 1y ago
  Those are arrays of integers, not of bytes. Most bytes are character-ish, which only python2's bytes acknowledged. Additionally, python2 supported a much richer set of operations on its bytes type than python3 does.

deleted 1y ago

[deleted]