3 ms·

Regex compilers know what a character is, so that "α+" doesn't match one 0xce followed by one-or-more 0xb1. Regex engines neither know nor care what a characte

Regex compilers know what a character is, so that "α+" doesn't match one 0xce followed by one-or-more 0xb1. Regex engines neither know nor care what a character is. They match bytes. Invalid UFT-8 is just another thing they won't match, along with all the valid sequences which also don't match.

camgunz 2y ago
> Regex engines neither know nor care what a character is. They match bytes. I think this is mostly not true, for example there's some interesting behavior in POSIX-compliant engines: "Other than POSIX-compliant engines part of a POSIX-compliant system, none of the regex flavors discussed in this tutorial support collating sequences. Note that a fully POSIX-compliant regex engine treats ch as a single character when the locale is set to Czech. This means that [^x]emie also matches chemie. [^x] matches a single character that is not an x, which includes ch in the Czech POSIX locale. In any other regular expression engine, or in a POSIX engine using a locale that does not treat ch as a digraph, [^x]emie matches the misspelled word cemie but not chemie, as [^x] cannot match the two characters ch." If you were just matching bytes you wouldn't do any of this. These engines clearly know that "ch" is two characters and two bytes under some locales and 1 character and two bytes under others. Same stuff goes for the Unicode regexes: using the traditional patterns leads to unexpected results: "In Unicode, à can be encoded as two code points: U+0061 (a) followed by U+0300 (grave accent). In this situation, . applied to à will match a without the accent. ^.$ will fail to match, since the string consists of two code points. ^..$ matches à." [1]. \p{Letter} works correctly here though, again showing that these engines know about more than just bytes. [0]: https://www.regular-expressions.info/posixbrackets.html#coll https://www.regular-expressions.info/posixbrackets.html#coll [1]: https://www.regular-expressions.info/unicode.html https://www.regular-expressions.info/unicode.html
zzo38computer 2y ago
The regex compiler then can have a flag to treat the regex as Unicode or non-Unicode. The regex engine then does not need to care which flag the compiler used.

camgunz 2y ago

> Regex engines neither know nor care what a character is. They match bytes. I think this is mostly not true, for example there's some interesting behavior in POSIX-compliant engines: "Other than POSIX-compliant engines part of a POSIX-compliant system, none of the regex flavors discussed in this tutorial support collating sequences. Note that a fully POSIX-compliant regex engine treats ch as a single character when the locale is set to Czech. This means that [^x]emie also matches chemie. [^x] matches a single character that is not an x, which includes ch in the Czech POSIX locale. In any other regular expression engine, or in a POSIX engine using a locale that does not treat ch as a digraph, [^x]emie matches the misspelled word cemie but not chemie, as [^x] cannot match the two characters ch." If you were just matching bytes you wouldn't do any of this. These engines clearly know that "ch" is two characters and two bytes under some locales and 1 character and two bytes under others. Same stuff goes for the Unicode regexes: using the traditional patterns leads to unexpected results: "In Unicode, à can be encoded as two code points: U+0061 (a) followed by U+0300 (grave accent). In this situation, . applied to à will match a without the accent. ^.$ will fail to match, since the string consists of two code points. ^..$ matches à." [1]. \p{Letter} works correctly here though, again showing that these engines know about more than just bytes. [0]: https://www.regular-expressions.info/posixbrackets.html#coll https://www.regular-expressions.info/posixbrackets.html#coll [1]: https://www.regular-expressions.info/unicode.html https://www.regular-expressions.info/unicode.html

zzo38computer 2y ago

The regex compiler then can have a flag to treat the regex as Unicode or non-Unicode. The regex engine then does not need to care which flag the compiler used.