3 ms·

Since you mentioned being a regex implementor, could you give me your opinion of the PCRE2 library? I had always assumed it to be a well polished mature library

Since you mentioned being a regex implementor, could you give me your opinion of the PCRE2 library? I had always assumed it to be a well polished mature library (simply because it's popular), but learnt this week that it doesn't support even basic Unicode properties like `\p{Alphabetic}`. According to the linked article, it wouldn't pass Level 1.2 Unicode support. A lot of code points used in Indic languages are outside the `L` categories, and not having the Alphabetic property (and having both `\w` and the POSIX `:alpha:` match only within `L`) is a significant annoyance when parsing mixed-language text (in Julia, which uses PCRE2 as the backend for regex). Is PCRE2 particularly behind in Unicode property implementation, or is this par for the course in common regex engines today?

burntsushi 8y ago
At a high level, PCRE2 is one of the most mature and fastest regex implementations out there (among ones that are open and that I know about). w.r.t. to Unicode support, it is fairly par for the course. There are some regex implementations that have very good Unicode support. Python's `regex` library (not `re` in the standard library) is very comprehensive with Unicode, and I think the regex library that `icgrep` uses is also fairly comprehensive. Both of those extend well into level 2 of UTS#18. Rust's regex library (the one I wrote) has fairly complete level 1 support with a few hiccups, but that is a somewhat recent development. I imagine ICU's regex engine also has fairly complete UTS#18 support as well, although I've never used it myself. Similarly for Oniguruma. icgrep is the only one of the above that I mention that doesn't have a reputation for being slow (and is actually quite fast). But... its toolchain dependencies are considerable. PCRE2's Unicode support is pretty good, but definitely has its quirks. It definitely appears to support a limited set of properties, and appears mostly restricted to general categories and scripts, which gives a fair bit of breadth. You might be able to get close to \p{Alphabetic} by building its constituent parts: https://unicode.org/reports/tr44/#Alphabetic https://unicode.org/reports/tr44/#Alphabetic --- It looks like PCRE2 supports everything in Lowercase + Uppercase + Lt + Lm + Lo + Nl + Other_Alphabetic except for `Other_Alphabetic` (this is according to `man pcre2pattern`). Unfortunately, Other_Alphabetic is quite large, so it might be hard to just handjam that easily. Interestingly, with all that said, PCRE2 does support a Unicode interpretation of \w, which is defined by Unicode to be: http://unicode.org/reports/tr18/#word http://unicode.org/reports/tr18/#word \p{alpha} \p{gc=Mark} \p{digit} \p{gc=Connector_Punctuation} \p{Join_Control} which is a superset of \p{Alphabetic} (\p{alpha} is an alias for \p{alphabetic}). Note though that \w by default in PCRE2 is ASCII-only. You need to enable the UCP flag to get Unicode support. You mention Julia, and they've definitely had issues with PCRE2. For example, one thing that Rust's regex engine supports that PCRE2 doesn't is the ability to have Unicode features enabled while simultaneously searching possibly invalid UTF-8 without additional costly checks or undefined behavior. PCRE2 can't do that. See: https://lists.exim.org/lurker/message/20180412.091038.080cf6f2.en.html https://lists.exim.org/lurker/message/20180412.091038.080cf6... and https://github.com/JuliaLang/julia/pull/26731 https://github.com/JuliaLang/julia/pull/26731

burntsushi 8y ago

At a high level, PCRE2 is one of the most mature and fastest regex implementations out there (among ones that are open and that I know about). w.r.t. to Unicode support, it is fairly par for the course. There are some regex implementations that have very good Unicode support. Python's `regex` library (not `re` in the standard library) is very comprehensive with Unicode, and I think the regex library that `icgrep` uses is also fairly comprehensive. Both of those extend well into level 2 of UTS#18. Rust's regex library (the one I wrote) has fairly complete level 1 support with a few hiccups, but that is a somewhat recent development. I imagine ICU's regex engine also has fairly complete UTS#18 support as well, although I've never used it myself. Similarly for Oniguruma. icgrep is the only one of the above that I mention that doesn't have a reputation for being slow (and is actually quite fast). But... its toolchain dependencies are considerable. PCRE2's Unicode support is pretty good, but definitely has its quirks. It definitely appears to support a limited set of properties, and appears mostly restricted to general categories and scripts, which gives a fair bit of breadth. You might be able to get close to \p{Alphabetic} by building its constituent parts: https://unicode.org/reports/tr44/#Alphabetic https://unicode.org/reports/tr44/#Alphabetic --- It looks like PCRE2 supports everything in Lowercase + Uppercase + Lt + Lm + Lo + Nl + Other_Alphabetic except for `Other_Alphabetic` (this is according to `man pcre2pattern`). Unfortunately, Other_Alphabetic is quite large, so it might be hard to just handjam that easily. Interestingly, with all that said, PCRE2 does support a Unicode interpretation of \w, which is defined by Unicode to be: http://unicode.org/reports/tr18/#word http://unicode.org/reports/tr18/#word \p{alpha} \p{gc=Mark} \p{digit} \p{gc=Connector_Punctuation} \p{Join_Control} which is a superset of \p{Alphabetic} (\p{alpha} is an alias for \p{alphabetic}). Note though that \w by default in PCRE2 is ASCII-only. You need to enable the UCP flag to get Unicode support. You mention Julia, and they've definitely had issues with PCRE2. For example, one thing that Rust's regex engine supports that PCRE2 doesn't is the ability to have Unicode features enabled while simultaneously searching possibly invalid UTF-8 without additional costly checks or undefined behavior. PCRE2 can't do that. See: https://lists.exim.org/lurker/message/20180412.091038.080cf6f2.en.html https://lists.exim.org/lurker/message/20180412.091038.080cf6... and https://github.com/JuliaLang/julia/pull/26731 https://github.com/JuliaLang/julia/pull/26731