2 ms·
There are 3 types of case folding: 1. Simple one-to-one mappings -- E.g. `T` to `t`. These are typically the ones handled by `lower()` or similar methods as th
by rhdunn 1y ago
There are 3 types of case folding:
1. Simple one-to-one mappings -- E.g. `T` to `t`. These are typically the ones handled by `lower()` or similar methods as they work on single characters so can modify a string in place (the length of the string doesn't change).
2. More complex one-to-many mappings -- E.g. German `ß` to `ss`. These are covered by functions like `casefold()`. You can't modify the string in place so the function needs to always write to a new string buffer.
3. Locale-specific mappings -- This is what this bug is about. In Turkish `I` maps to `ı` whereas other languages/locales it maps to `i`. You can only implement this by passing the locale to the case function, irrespective of whether you are also doing (1) or (2).
- zahlman 1y agoThis is not quite right, at least for Python. .upper() and .lower() (and .casefold() as well) implement the default casing algorithms from the Unicode specification, which are one-to-many (but still locale-naive). Other languages, meanwhile, might well implement locale-aware mapping that defaults to the system locale rather than requiring a locale to be passed.
- rhdunn 1y agoFor one-to-one I was thinking of the Unicode Character Database single-character upper, lower, and title case entries in the UnicodeData.txt file [1] -- i.e. the simple case mappings [2]. The one-to-many mappings are specified in SpecialCasing.txt. Some are locale independent (like my German sharp/double S example) and others are locale aware (like the Turkish example). For the locale aware mappings you need to ensure you are setting the language/locale correctly in your language of choice, or that the language is doing the right thing. [1] https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.tx... [2] https://www.unicode.org/reports/tr44/#Casemapping https://www.unicode.org/reports/tr44/#Casemapping [3] https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing....