3 ms·

As a regex implementor, I am super appreciative of this document. I've consulted it quite a bit, and it helped guide me towards picking and choosing which Unico

As a regex implementor, I am super appreciative of this document. I've consulted it quite a bit, and it helped guide me towards picking and choosing which Unicode features I should support and what their semantics should be. The upshot is that many other regex engines have either inspired this document or been inspired by it, which gives at least some consistency between implementations. I also think it's clear from reading this document many times that it was very much informed by regex implementors. The text seems quite self-aware of the fact that some features may not be practical to implement in all scenarios. It is certainly far from perfect, and I bet that if we got in a room together, we could vent about things in (or missing from) this document. But I am determined not to let perfection be the enemy of good, and this document has, at least in my experience, done much more good than harm.

jonstewart 8y ago
Ditto to everything here. UTS #18 is a fantastic map for the implementor. I co-authored a paper about it: http://www.dfrws.org/sites/default/files/session-files/paper-unicode_search_of_dirty_data_or_how_i_learned_to_stop_worrying_and_love_unicode_technical_standard_18.pdf http://www.dfrws.org/sites/default/files/session-files/paper...
- glangdale 8y ago
  I still don't agree, although I respect both of your opinions as real regex implementors without a doubt. It's just too complex a 'standard'. The amount of crap they had to 'retract' by the current version is telling, and features like 3.9 and 3.11 are speculative design by standards committee, which is the exact opposite of how these things should work. It's the ultimate unfunded mandate: "hooray, I'm writing a standard, why don't I shove some random crap in there that no-one implements now in existing libraries in the hope I get to set directions". The standard should be minimal and well-defined. This is neither. It's also preposterously self-important. Yeah, I'm so fascinated by the history of the Unicode standard I'm going to start introducing an "age" property so I can regex-match on when characters were entered. There's a certain attitude whereby these guys are just blithely shoveling junk into a standard that no-one needs. If you're desperate for all these exotic character properties put them in a godamn file as a character class and blat them in wherever you need them (yes, it would be nice if regex had macros). Somewhere in all this nonsense is a good standard struggling to get out. They need to shit or get off the pot. Anything that they don't know how to do or that no-one knows how to do should be completely deleted. Anything that they offer 15 different choices as to how to do ... they need to figure out which is the best one and pick that. It's a standards documents, it's supposed to make choices, it's not a potted plant.
  - jonstewart 8y ago
    It has a bad case of featuritis; it reads both discursively and self-consciously, like it knows that there's really no justification for using regexps to specify named properties, but it just can't help itself. Some of the Level 3 features venture too far afield from Unicode, too, e.g., 3.11 is "implement YACC" and has nothing to do with Unicode and 3.7 is "implement a streaming regexp engine, we dare you" (challenge accepted, Unicode dweebs!). So, it is a horrible standard by which to judge overall conformance, but no one really does that. I think the value is in lighting the way through a broad array of thorny Unicode topics. And I think that the implementor who grapples with a majority subset of the sensible topics will wind up with an engine that handles users' Unicode needs.
sundarurfriend 8y ago
Since you mentioned being a regex implementor, could you give me your opinion of the PCRE2 library? I had always assumed it to be a well polished mature library (simply because it's popular), but learnt this week that it doesn't support even basic Unicode properties like `\p{Alphabetic}`. According to the linked article, it wouldn't pass Level 1.2 Unicode support. A lot of code points used in Indic languages are outside the `L` categories, and not having the Alphabetic property (and having both `\w` and the POSIX `:alpha:` match only within `L`) is a significant annoyance when parsing mixed-language text (in Julia, which uses PCRE2 as the backend for regex). Is PCRE2 particularly behind in Unicode property implementation, or is this par for the course in common regex engines today?
- burntsushi 8y ago
  At a high level, PCRE2 is one of the most mature and fastest regex implementations out there (among ones that are open and that I know about). w.r.t. to Unicode support, it is fairly par for the course. There are some regex implementations that have very good Unicode support. Python's `regex` library (not `re` in the standard library) is very comprehensive with Unicode, and I think the regex library that `icgrep` uses is also fairly comprehensive. Both of those extend well into level 2 of UTS#18. Rust's regex library (the one I wrote) has fairly complete level 1 support with a few hiccups, but that is a somewhat recent development. I imagine ICU's regex engine also has fairly complete UTS#18 support as well, although I've never used it myself. Similarly for Oniguruma. icgrep is the only one of the above that I mention that doesn't have a reputation for being slow (and is actually quite fast). But... its toolchain dependencies are considerable. PCRE2's Unicode support is pretty good, but definitely has its quirks. It definitely appears to support a limited set of properties, and appears mostly restricted to general categories and scripts, which gives a fair bit of breadth. You might be able to get close to \p{Alphabetic} by building its constituent parts: https://unicode.org/reports/tr44/#Alphabetic https://unicode.org/reports/tr44/#Alphabetic --- It looks like PCRE2 supports everything in Lowercase + Uppercase + Lt + Lm + Lo + Nl + Other_Alphabetic except for `Other_Alphabetic` (this is according to `man pcre2pattern`). Unfortunately, Other_Alphabetic is quite large, so it might be hard to just handjam that easily. Interestingly, with all that said, PCRE2 does support a Unicode interpretation of \w, which is defined by Unicode to be: http://unicode.org/reports/tr18/#word http://unicode.org/reports/tr18/#word \p{alpha} \p{gc=Mark} \p{digit} \p{gc=Connector_Punctuation} \p{Join_Control} which is a superset of \p{Alphabetic} (\p{alpha} is an alias for \p{alphabetic}). Note though that \w by default in PCRE2 is ASCII-only. You need to enable the UCP flag to get Unicode support. You mention Julia, and they've definitely had issues with PCRE2. For example, one thing that Rust's regex engine supports that PCRE2 doesn't is the ability to have Unicode features enabled while simultaneously searching possibly invalid UTF-8 without additional costly checks or undefined behavior. PCRE2 can't do that. See: https://lists.exim.org/lurker/message/20180412.091038.080cf6f2.en.html https://lists.exim.org/lurker/message/20180412.091038.080cf6... and https://github.com/JuliaLang/julia/pull/26731 https://github.com/JuliaLang/julia/pull/26731

jonstewart 8y ago

Ditto to everything here. UTS #18 is a fantastic map for the implementor. I co-authored a paper about it: http://www.dfrws.org/sites/default/files/session-files/paper-unicode_search_of_dirty_data_or_how_i_learned_to_stop_worrying_and_love_unicode_technical_standard_18.pdf http://www.dfrws.org/sites/default/files/session-files/paper...

glangdale 8y ago
I still don't agree, although I respect both of your opinions as real regex implementors without a doubt. It's just too complex a 'standard'. The amount of crap they had to 'retract' by the current version is telling, and features like 3.9 and 3.11 are speculative design by standards committee, which is the exact opposite of how these things should work. It's the ultimate unfunded mandate: "hooray, I'm writing a standard, why don't I shove some random crap in there that no-one implements now in existing libraries in the hope I get to set directions". The standard should be minimal and well-defined. This is neither. It's also preposterously self-important. Yeah, I'm so fascinated by the history of the Unicode standard I'm going to start introducing an "age" property so I can regex-match on when characters were entered. There's a certain attitude whereby these guys are just blithely shoveling junk into a standard that no-one needs. If you're desperate for all these exotic character properties put them in a godamn file as a character class and blat them in wherever you need them (yes, it would be nice if regex had macros). Somewhere in all this nonsense is a good standard struggling to get out. They need to shit or get off the pot. Anything that they don't know how to do or that no-one knows how to do should be completely deleted. Anything that they offer 15 different choices as to how to do ... they need to figure out which is the best one and pick that. It's a standards documents, it's supposed to make choices, it's not a potted plant.
- jonstewart 8y ago
  It has a bad case of featuritis; it reads both discursively and self-consciously, like it knows that there's really no justification for using regexps to specify named properties, but it just can't help itself. Some of the Level 3 features venture too far afield from Unicode, too, e.g., 3.11 is "implement YACC" and has nothing to do with Unicode and 3.7 is "implement a streaming regexp engine, we dare you" (challenge accepted, Unicode dweebs!). So, it is a horrible standard by which to judge overall conformance, but no one really does that. I think the value is in lighting the way through a broad array of thorny Unicode topics. And I think that the implementor who grapples with a majority subset of the sensible topics will wind up with an engine that handles users' Unicode needs.

sundarurfriend 8y ago

Since you mentioned being a regex implementor, could you give me your opinion of the PCRE2 library? I had always assumed it to be a well polished mature library (simply because it's popular), but learnt this week that it doesn't support even basic Unicode properties like `\p{Alphabetic}`. According to the linked article, it wouldn't pass Level 1.2 Unicode support. A lot of code points used in Indic languages are outside the `L` categories, and not having the Alphabetic property (and having both `\w` and the POSIX `:alpha:` match only within `L`) is a significant annoyance when parsing mixed-language text (in Julia, which uses PCRE2 as the backend for regex). Is PCRE2 particularly behind in Unicode property implementation, or is this par for the course in common regex engines today?

burntsushi 8y ago
At a high level, PCRE2 is one of the most mature and fastest regex implementations out there (among ones that are open and that I know about). w.r.t. to Unicode support, it is fairly par for the course. There are some regex implementations that have very good Unicode support. Python's `regex` library (not `re` in the standard library) is very comprehensive with Unicode, and I think the regex library that `icgrep` uses is also fairly comprehensive. Both of those extend well into level 2 of UTS#18. Rust's regex library (the one I wrote) has fairly complete level 1 support with a few hiccups, but that is a somewhat recent development. I imagine ICU's regex engine also has fairly complete UTS#18 support as well, although I've never used it myself. Similarly for Oniguruma. icgrep is the only one of the above that I mention that doesn't have a reputation for being slow (and is actually quite fast). But... its toolchain dependencies are considerable. PCRE2's Unicode support is pretty good, but definitely has its quirks. It definitely appears to support a limited set of properties, and appears mostly restricted to general categories and scripts, which gives a fair bit of breadth. You might be able to get close to \p{Alphabetic} by building its constituent parts: https://unicode.org/reports/tr44/#Alphabetic https://unicode.org/reports/tr44/#Alphabetic --- It looks like PCRE2 supports everything in Lowercase + Uppercase + Lt + Lm + Lo + Nl + Other_Alphabetic except for `Other_Alphabetic` (this is according to `man pcre2pattern`). Unfortunately, Other_Alphabetic is quite large, so it might be hard to just handjam that easily. Interestingly, with all that said, PCRE2 does support a Unicode interpretation of \w, which is defined by Unicode to be: http://unicode.org/reports/tr18/#word http://unicode.org/reports/tr18/#word \p{alpha} \p{gc=Mark} \p{digit} \p{gc=Connector_Punctuation} \p{Join_Control} which is a superset of \p{Alphabetic} (\p{alpha} is an alias for \p{alphabetic}). Note though that \w by default in PCRE2 is ASCII-only. You need to enable the UCP flag to get Unicode support. You mention Julia, and they've definitely had issues with PCRE2. For example, one thing that Rust's regex engine supports that PCRE2 doesn't is the ability to have Unicode features enabled while simultaneously searching possibly invalid UTF-8 without additional costly checks or undefined behavior. PCRE2 can't do that. See: https://lists.exim.org/lurker/message/20180412.091038.080cf6f2.en.html https://lists.exim.org/lurker/message/20180412.091038.080cf6... and https://github.com/JuliaLang/julia/pull/26731 https://github.com/JuliaLang/julia/pull/26731