9 ms·

Unrelated story about Russian language. The first letter of russian alphabet is А, the last one is Я. So it's natural to try to match russian words with '[А-Яа

Unrelated story about Russian language. The first letter of russian alphabet is А, the last one is Я. So it's natural to try to match russian words with '[А-Яа-я]+'. But this is a recipe for disaster, this regexp doesn't match words with 'Ё' in them like "Артём". This is due to the fact that regexp ranges work on byte values. All letters of russian language have neatly ordered byte values, except for the Ё.

blkhawk 6y ago
not that unusual - for German for instance üöäÜÖÄß need to be added so all words can be matched.
- a3w 6y ago
  now, there is even a capital ß ;)
  - Forge36 6y ago
    Out of curiosity I tried on my phone: ß Ss SS So my phone doesn't have that yet!
    - maxpro 6y ago
      mine has it - ẞ the small one is ß
      - Igelau 6y ago
        Is the capital supposed to be shorter?
        zaarn 6y ago
        It's wider. Depends on your font and it's support though.
        a1369209993 6y ago
        Oddly enough, yes (depending on font of course); compare "Sf": "ẞß" should have the same capital and lowercase-ascender heights, the latter of which is often higher.
  - StavrosK 6y ago
    Less relatedly, I really hate when people use the eszett instead of a Greek beta. I just needed to get that out of my chest.
mikepurvis 6y ago
This would be an argument for just using [:alpha:] everywhere; presumably it does the correct thing based on locale?
- tryauuum 6y ago
  No, alpha doesn't work, at least in "grep -P" with "ru_RU.UTF-8" locale: $ echo Test | grep -oP '[[:alpha:]]+' Test $ echo Артём | grep -oP '[[:alpha:]]+' $ echo Артём | grep -oP '[А-Яа-я]+' Арт м This thing works, though I've never seen one in the wild: $ echo Артём | grep -oP '[\p{Cyrillic}]+' Артём
  - wruza 6y ago
    Standard classes only work for 8-bit locales, afaik, and also in some languages (e.g. perl) when a string encoding corresponds to an internal representation of what its engine thinks it is a "current" unicode format for the specific version of language and a checkout location. The fact that different tools stick to different engines, modes and normalization rules (bre/ere/posix/nfd/nfc/perl/pcre/php/ruby/icu/whatever) doesn't help either. Full cross-platform unicode matching is a can of worms that you usually don't want to open. It is basically CSV of the encoding world. /\S+?/ to the rescue. https://regular-expressions.mobi/unicode.html https://regular-expressions.mobi/unicode.html https://regular-expressions.mobi/refunicode.html https://regular-expressions.mobi/refunicode.html https://regular-expressions.mobi/posixbrackets.html https://regular-expressions.mobi/posixbrackets.html
    - mikepurvis 6y ago
      Well, that is horrifying. TIL.
  - drran 6y ago
    $ echo Артём | grep -oE '[[:alpha:]]+' Артём
    - tryauuum 6y ago
Sharlin 6y ago
English is probably the only commonly spoken language where naïve char range matching kind of sort of works. I say ”kind of sort of” because [a-zA-Z] trivially fails to match all words in many English texts that haven’t been lossily compressed to ASCII, including this comment. It is practically always wrong to match on [a-z] unless you’re parsing a computer language whose spec guarantees that it works.
- tryauuum 6y ago
  I always wanted to know, how easy is it to type naïve on a common western keyboard? Do you have to press some obscure keyboard shortcut?
  - dylz 6y ago
    "i or similar works. ^i, `i, 'i, etc. for the others.
  - reaperducer 6y ago
    how easy is it to type naïve on a common western keyboard? In macOS, you can either use Command-u (for "umlat") followed by i, or hold down the i key for a second and press 2 to select the ï from the pop-up menu.
    - masklinn 6y ago
      > Command-u option-u (aka alt-u). Generally speaking, command is for application-level or os-level commands, control is for text edition, and alt is for alternate characters (all can be shifted and command "overrides" the rest).
      - reaperducer 6y ago
        You're right, it's Option-u. Most of the key labels on my MacBook have long since been scratched away. This has happened with every single Apple keyboard I've ever used. I suspect it's my fault, since I'm a key pounder, having learned to type on an IBM Selectric typewriter.
noxer 6y ago
The problem here is simply a bad regex pattern. You need to use something that supports Unicode an enable it. (Usually a flag or option you have to set because Unicode matching is slower and often not needed.) Then last but not least you need to use character classes and not define ranges yourself unless you actually need a range. Like you cant use [:digit:] or \d if you really only want 1-9 without the 0. Here are some examples to match your string all of them use PCRE flavor (UCP)[[:alpha:]]+ (UCP)[[:alnum:]]+ This would includes digits (UCP)[[:word:]]+ This includes "word" chars (UCP)\w+ same as above

blkhawk 6y ago

not that unusual - for German for instance üöäÜÖÄß need to be added so all words can be matched.

a3w 6y ago
now, there is even a capital ß ;)
- Forge36 6y ago
  Out of curiosity I tried on my phone: ß Ss SS So my phone doesn't have that yet!
  - maxpro 6y ago
    mine has it - ẞ the small one is ß
    - Igelau 6y ago
      Is the capital supposed to be shorter?
      - zaarn 6y ago
        It's wider. Depends on your font and it's support though.
      - a1369209993 6y ago
        Oddly enough, yes (depending on font of course); compare "Sf": "ẞß" should have the same capital and lowercase-ascender heights, the latter of which is often higher.
- StavrosK 6y ago
  Less relatedly, I really hate when people use the eszett instead of a Greek beta. I just needed to get that out of my chest.

mikepurvis 6y ago

This would be an argument for just using [:alpha:] everywhere; presumably it does the correct thing based on locale?

tryauuum 6y ago
No, alpha doesn't work, at least in "grep -P" with "ru_RU.UTF-8" locale: $ echo Test | grep -oP '[[:alpha:]]+' Test $ echo Артём | grep -oP '[[:alpha:]]+' $ echo Артём | grep -oP '[А-Яа-я]+' Арт м This thing works, though I've never seen one in the wild: $ echo Артём | grep -oP '[\p{Cyrillic}]+' Артём
- wruza 6y ago
  Standard classes only work for 8-bit locales, afaik, and also in some languages (e.g. perl) when a string encoding corresponds to an internal representation of what its engine thinks it is a "current" unicode format for the specific version of language and a checkout location. The fact that different tools stick to different engines, modes and normalization rules (bre/ere/posix/nfd/nfc/perl/pcre/php/ruby/icu/whatever) doesn't help either. Full cross-platform unicode matching is a can of worms that you usually don't want to open. It is basically CSV of the encoding world. /\S+?/ to the rescue. https://regular-expressions.mobi/unicode.html https://regular-expressions.mobi/unicode.html https://regular-expressions.mobi/refunicode.html https://regular-expressions.mobi/refunicode.html https://regular-expressions.mobi/posixbrackets.html https://regular-expressions.mobi/posixbrackets.html
  - mikepurvis 6y ago
    Well, that is horrifying. TIL.
- drran 6y ago
  $ echo Артём | grep -oE '[[:alpha:]]+' Артём
  - tryauuum 6y ago

Sharlin 6y ago

English is probably the only commonly spoken language where naïve char range matching kind of sort of works. I say ”kind of sort of” because [a-zA-Z] trivially fails to match all words in many English texts that haven’t been lossily compressed to ASCII, including this comment. It is practically always wrong to match on [a-z] unless you’re parsing a computer language whose spec guarantees that it works.

tryauuum 6y ago
I always wanted to know, how easy is it to type naïve on a common western keyboard? Do you have to press some obscure keyboard shortcut?
- dylz 6y ago
  "i or similar works. ^i, `i, 'i, etc. for the others.
- reaperducer 6y ago
  how easy is it to type naïve on a common western keyboard? In macOS, you can either use Command-u (for "umlat") followed by i, or hold down the i key for a second and press 2 to select the ï from the pop-up menu.
  - masklinn 6y ago
    > Command-u option-u (aka alt-u). Generally speaking, command is for application-level or os-level commands, control is for text edition, and alt is for alternate characters (all can be shifted and command "overrides" the rest).
    - reaperducer 6y ago
      You're right, it's Option-u. Most of the key labels on my MacBook have long since been scratched away. This has happened with every single Apple keyboard I've ever used. I suspect it's my fault, since I'm a key pounder, having learned to type on an IBM Selectric typewriter.

noxer 6y ago

The problem here is simply a bad regex pattern. You need to use something that supports Unicode an enable it. (Usually a flag or option you have to set because Unicode matching is slower and often not needed.) Then last but not least you need to use character classes and not define ranges yourself unless you actually need a range. Like you cant use [:digit:] or \d if you really only want 1-9 without the 0. Here are some examples to match your string all of them use PCRE flavor (UCP)[[:alpha:]]+ (UCP)[[:alnum:]]+ This would includes digits (UCP)[[:word:]]+ This includes "word" chars (UCP)\w+ same as above