3 ms·
This is true, almost all "string" operations require you to know extra information about the string's "type"—which, yes, includes its encoding. Python 3 takes
by naniwaduni 4y ago
This is true, almost all "string" operations require you to know extra information about the string's "type"—which, yes, includes its encoding.
Python 3 takes this fact and runs with the stance that strings should, by default, carry their encoding with them (and that encoding is specifically, more or less, a sequence of quasi-Unicode code points with unspecified internal representation). But this is an intensely half-baked solution to the wrong problem, because you need to know much more about a string than its encoding for just about any string operation you'd want to do. Just with Unicode code point concatenation, you can run into plenty of trouble with:
- strings containing control characters
- strings with unpaired surrogates (which okay, shouldn't really be there, but get used for various reasons anyway)
- strings containing combining characters
- strings in different natural languages
- strings in languages with ligatures
- strings with ligature-like emoji
(and this is obviously not remotely close to exhaustive.)