4 ms·
> ‘["-/]’ is perfectly valid in ASCII, but is not valid in many Unicode locales, such as en_US.UTF-8. Why is this the case? Collation sequences, I’m guessing?
by chrismorgan 8y ago
> ‘["-/]’ is perfectly valid in ASCII, but is not valid in many Unicode locales, such as en_US.UTF-8.
Why is this the case? Collation sequences, I’m guessing?
- jwilk 8y agoYes, slash sorts before double-quote in this locale: $ (echo '"'; echo '/') | LC_ALL=en_US.UTF-8 sort / "
- a1369209993 8y agoEdit: nevermind, I apparently rm'd the offending locale directory last time I encountered a bug like this and sort is silently ignoring LC_ALL: bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory I can't reproduce this bug: $ (echo '"'; echo '/') | LC_ALL=en_US.UTF-8 sort " / $ sort --version sort (GNU coreutils) 8.26 What version are you using?
- raverbashing 8y agoWell, that's weird, I though Unicode matched ASCII for codepoints < 127 Still this doesn't look like an issue with regex's per se
- LukeShu 8y agoThe numeric value of Unicode codepoints does match ASCII for codepoints < 127. However, Unicode codepoint numeric order is not collation order. Collation order isn't encoding-specific, it's locale (language)-specific. My current locale is en_US.UTF-8; if I changed that to a different locale ending with ".UTF-8", the codepoint values haven't changed (still Unicode), and the encodings of them haven't changed (still UTF-8), but the collation order might have. One can observe this change even without getting exotic. In the "C" or "POSIX" locales (synonyms), the ASCII values sort in ASCII numeric order ("ABC..XYZ...abc...xyz"), but in "en_US", they sort "aAbBcC...xXyYzZ" (because that's what a native American English speaker would expect, if they're not familiar with ASCII).
- chrismorgan 8y ago… ouch. So if you’re operating by locale, [a-z] and [A-Z] may do the wrong thing completely, including all but the first or last letter of the wrong case. Putting lowercase before uppercase also surprises me; if asked to sort them, I’d definitely put uppercases before lowercases: AaBbCc…
- LukeShu 8y agoYes, that's why people found GNU awk <4.0 troublesome. Fortunately modern versions POSIX make regex range expressions ("[a-z]") unspecified behavior on locales other than "C", so modern versions of GNU awk and other GNU utilities "do the right thing". As for the en_US example, for the most part collation is case-insensitive, and case only comes in as a tiebreaker; "aa" < "Aa" < "ab".