3 ms·
At a high level, PCRE2 is one of the most mature and fastest regex implementations out there (among ones that are open and that I know about). w.r.t. to Unicod
by burntsushi 8y ago
At a high level, PCRE2 is one of the most mature and fastest regex implementations out there (among ones that are open and that I know about).
w.r.t. to Unicode support, it is fairly par for the course. There are some regex implementations that have very good Unicode support. Python's `regex` library (not `re` in the standard library) is very comprehensive with Unicode, and I think the regex library that `icgrep` uses is also fairly comprehensive. Both of those extend well into level 2 of UTS#18. Rust's regex library (the one I wrote) has fairly complete level 1 support with a few hiccups, but that is a somewhat recent development.
I imagine ICU's regex engine also has fairly complete UTS#18 support as well, although I've never used it myself. Similarly for Oniguruma.
icgrep is the only one of the above that I mention that doesn't have a reputation for being slow (and is actually quite fast). But... its toolchain dependencies are considerable.
PCRE2's Unicode support is pretty good, but definitely has its quirks. It definitely appears to support a limited set of properties, and appears mostly restricted to general categories and scripts, which gives a fair bit of breadth. You might be able to get close to \p{Alphabetic} by building its constituent parts: https://unicode.org/reports/tr44/#Alphabetic https://unicode.org/reports/tr44/#Alphabetic --- It looks like PCRE2 supports everything in Lowercase + Uppercase + Lt + Lm + Lo + Nl + Other_Alphabetic except for `Other_Alphabetic` (this is according to `man pcre2pattern`). Unfortunately, Other_Alphabetic is quite large, so it might be hard to just handjam that easily.
Interestingly, with all that said, PCRE2 does support a Unicode interpretation of \w, which is defined by Unicode to be: http://unicode.org/reports/tr18/#word http://unicode.org/reports/tr18/#word
\p{alpha}
\p{gc=Mark}
\p{digit}
\p{gc=Connector_Punctuation}
\p{Join_Control}
which is a superset of \p{Alphabetic} (\p{alpha} is an alias for \p{alphabetic}). Note though that \w by default in PCRE2 is ASCII-only. You need to enable the UCP flag to get Unicode support.
You mention Julia, and they've definitely had issues with PCRE2. For example, one thing that Rust's regex engine supports that PCRE2 doesn't is the ability to have Unicode features enabled while simultaneously searching possibly invalid UTF-8 without additional costly checks or undefined behavior. PCRE2 can't do that. See: https://lists.exim.org/lurker/message/20180412.091038.080cf6f2.en.html https://lists.exim.org/lurker/message/20180412.091038.080cf6... and https://github.com/JuliaLang/julia/pull/26731 https://github.com/JuliaLang/julia/pull/26731