3 ms·
The numeric value of Unicode codepoints does match ASCII for codepoints < 127. However, Unicode codepoint numeric order is not collation order. Collation orde
by LukeShu 8y ago
The numeric value of Unicode codepoints does match ASCII for codepoints < 127. However, Unicode codepoint numeric order is not collation order. Collation order isn't encoding-specific, it's locale (language)-specific.
My current locale is en_US.UTF-8; if I changed that to a different locale ending with ".UTF-8", the codepoint values haven't changed (still Unicode), and the encodings of them haven't changed (still UTF-8), but the collation order might have.
One can observe this change even without getting exotic. In the "C" or "POSIX" locales (synonyms), the ASCII values sort in ASCII numeric order ("ABC..XYZ...abc...xyz"), but in "en_US", they sort "aAbBcC...xXyYzZ" (because that's what a native American English speaker would expect, if they're not familiar with ASCII).
- chrismorgan 8y ago… ouch. So if you’re operating by locale, [a-z] and [A-Z] may do the wrong thing completely, including all but the first or last letter of the wrong case. Putting lowercase before uppercase also surprises me; if asked to sort them, I’d definitely put uppercases before lowercases: AaBbCc…
- LukeShu 8y agoYes, that's why people found GNU awk <4.0 troublesome. Fortunately modern versions POSIX make regex range expressions ("[a-z]") unspecified behavior on locales other than "C", so modern versions of GNU awk and other GNU utilities "do the right thing". As for the en_US example, for the most part collation is case-insensitive, and case only comes in as a tiebreaker; "aa" < "Aa" < "ab".