6 ms·
Python's strings have uppercase, lowercase and case-folding methods that don't choke on this. They don't use UTF-16 internally (they can use UCS-2 for strings w
by zahlman 2y ago
Python's strings have uppercase, lowercase and case-folding methods that don't choke on this. They don't use UTF-16 internally (they can use UCS-2 for strings whose code points will fit in that range; while a string might store code points from the surrogate-pair range, they're never interpreted as surrogate pairs, but instead as an error encoding so that e.g. invalid UTF-8 can be round-tripped) so they're never worried about surrogate pairs, and it knows a few things about localized text casing:
>>> 'ß'.upper()
'SS'
>>> 'ß'.lower()
'ß'
>>> 'ß'.casefold()
'ss'
There are a lot of really complicated tasks for Unicode strings. String casing isn't really one of them.
(No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.)
- kccqzy 2y agoStill breaks on, for example, Turkish i vs İ. It's impossible to do correctly without language information. > (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.) Yes that's my point. Because in typical languages strings don't store language metadata, this is impossible to do correctly in general.
- zahlman 2y agoI'm not seeing anything in the Swift documentation about strings carrying language metadata, either, though?
- kccqzy 2y agoThis lowercase function takes a locale argument https://developer.apple.com/documentation/foundation/nsstring/1417298-lowercased https://developer.apple.com/documentation/foundation/nsstrin... It looks like an old NSString method that's available in both Obj-C and Swift. The casefold function is even older than that. https://developer.apple.com/documentation/foundation/nsstring/1413779-folding https://developer.apple.com/documentation/foundation/nsstrin... Its documentation specifically includes a discussion of the Turkish İ/I issue.
- tedunangst 2y agoBut that's wrong. The upper case for ß is ẞ.
- IncreasePosts 2y agoThat was only adopted in Germany like 7 years ago!
- kccqzy 2y agoWell languages and conventions change. The € sign was added not that long ago and it was somewhat painful. The Chinese language uses a single character to refer to chemical elements so when IUPAC names new elements they will invent new characters. Etc.
- extraduder_ire 2y agoDoes unicode have space set aside for those new symbols to slot into? I know it's very rare, but it could get messy.
- account42 2y agoUnicode is already messy. Chinese characters especially so due to han unificiation.
- cm2187 2y agoC#'s "ToUpper" takes an optional CultureInfo argument if you want to play around with how to treat different languages. Again, solved problem decades ago.
- account42 2y agoThis is not a locale issue, it's a Unicode version issue. Which hightlights another problem with adding this to the base standard library.
- 2y ago
- crote 2y agoBut that's wrong. The uppercase for "in Maßen" ("in moderate amounts") is not "IN MASSEN" ("in Massen", meaning "in massive amounts").