4 ms·

This is not really a standard; nearly any behavior which is complicated is left undefined or expressed with vague, aspirational language ("an implementation sho

This is not really a standard; nearly any behavior which is complicated is left undefined or expressed with vague, aspirational language ("an implementation should do this somehow, I dunno"). It doesn't look like regex implementers were consulted at any time.

Someone 8y ago
“Should” likely should be interpreted as defined in https://www.ietf.org/rfc/rfc2119.txt https://www.ietf.org/rfc/rfc2119.txt. I think there are a lot of should’s because the Unicode consortium realizes that there will be variability in feature sets of regular expression libraries. http://unicode.org/reports/tr18/#Conformance http://unicode.org/reports/tr18/#Conformance has quite a few shall’s to counteract that, but still leaves implementations to pick and choose features, as long as they document that.
- jwilk 8y ago
  HTML rendering of said RFC: https://tools.ietf.org/html/rfc2119 https://tools.ietf.org/html/rfc2119
garmaine 8y ago
“Should” has a very precise technical meaning in standards. Mentally it’s best to replace it with “MUST” (they are equivalent).
- quicklyfrozen 8y ago
  "Must" is equivalent to required, "should" is equivalent to recommended, so not at all the same.
  - garmaine 8y ago
    True enough. I was conflating “should” with “shall” which is definitionally equal to “must”. The edit window has elapsed or I’d correct it.
- glangdale 8y ago
  True enough; however, when the behavior after the should isn't properly specified...
burntsushi 8y ago
As a regex implementor, I am super appreciative of this document. I've consulted it quite a bit, and it helped guide me towards picking and choosing which Unicode features I should support and what their semantics should be. The upshot is that many other regex engines have either inspired this document or been inspired by it, which gives at least some consistency between implementations. I also think it's clear from reading this document many times that it was very much informed by regex implementors. The text seems quite self-aware of the fact that some features may not be practical to implement in all scenarios. It is certainly far from perfect, and I bet that if we got in a room together, we could vent about things in (or missing from) this document. But I am determined not to let perfection be the enemy of good, and this document has, at least in my experience, done much more good than harm.
- jonstewart 8y ago
  Ditto to everything here. UTS #18 is a fantastic map for the implementor. I co-authored a paper about it: http://www.dfrws.org/sites/default/files/session-files/paper-unicode_search_of_dirty_data_or_how_i_learned_to_stop_worrying_and_love_unicode_technical_standard_18.pdf http://www.dfrws.org/sites/default/files/session-files/paper...
  - glangdale 8y ago
    I still don't agree, although I respect both of your opinions as real regex implementors without a doubt. It's just too complex a 'standard'. The amount of crap they had to 'retract' by the current version is telling, and features like 3.9 and 3.11 are speculative design by standards committee, which is the exact opposite of how these things should work. It's the ultimate unfunded mandate: "hooray, I'm writing a standard, why don't I shove some random crap in there that no-one implements now in existing libraries in the hope I get to set directions". The standard should be minimal and well-defined. This is neither. It's also preposterously self-important. Yeah, I'm so fascinated by the history of the Unicode standard I'm going to start introducing an "age" property so I can regex-match on when characters were entered. There's a certain attitude whereby these guys are just blithely shoveling junk into a standard that no-one needs. If you're desperate for all these exotic character properties put them in a godamn file as a character class and blat them in wherever you need them (yes, it would be nice if regex had macros). Somewhere in all this nonsense is a good standard struggling to get out. They need to shit or get off the pot. Anything that they don't know how to do or that no-one knows how to do should be completely deleted. Anything that they offer 15 different choices as to how to do ... they need to figure out which is the best one and pick that. It's a standards documents, it's supposed to make choices, it's not a potted plant.

Someone 8y ago

“Should” likely should be interpreted as defined in https://www.ietf.org/rfc/rfc2119.txt https://www.ietf.org/rfc/rfc2119.txt. I think there are a lot of should’s because the Unicode consortium realizes that there will be variability in feature sets of regular expression libraries. http://unicode.org/reports/tr18/#Conformance http://unicode.org/reports/tr18/#Conformance has quite a few shall’s to counteract that, but still leaves implementations to pick and choose features, as long as they document that.

jwilk 8y ago
HTML rendering of said RFC: https://tools.ietf.org/html/rfc2119 https://tools.ietf.org/html/rfc2119

garmaine 8y ago

“Should” has a very precise technical meaning in standards. Mentally it’s best to replace it with “MUST” (they are equivalent).

quicklyfrozen 8y ago
"Must" is equivalent to required, "should" is equivalent to recommended, so not at all the same.
- garmaine 8y ago
  True enough. I was conflating “should” with “shall” which is definitionally equal to “must”. The edit window has elapsed or I’d correct it.
glangdale 8y ago
True enough; however, when the behavior after the should isn't properly specified...

burntsushi 8y ago

As a regex implementor, I am super appreciative of this document. I've consulted it quite a bit, and it helped guide me towards picking and choosing which Unicode features I should support and what their semantics should be. The upshot is that many other regex engines have either inspired this document or been inspired by it, which gives at least some consistency between implementations. I also think it's clear from reading this document many times that it was very much informed by regex implementors. The text seems quite self-aware of the fact that some features may not be practical to implement in all scenarios. It is certainly far from perfect, and I bet that if we got in a room together, we could vent about things in (or missing from) this document. But I am determined not to let perfection be the enemy of good, and this document has, at least in my experience, done much more good than harm.

jonstewart 8y ago
Ditto to everything here. UTS #18 is a fantastic map for the implementor. I co-authored a paper about it: http://www.dfrws.org/sites/default/files/session-files/paper-unicode_search_of_dirty_data_or_how_i_learned_to_stop_worrying_and_love_unicode_technical_standard_18.pdf http://www.dfrws.org/sites/default/files/session-files/paper...
- glangdale 8y ago
  I still don't agree, although I respect both of your opinions as real regex implementors without a doubt. It's just too complex a 'standard'. The amount of crap they had to 'retract' by the current version is telling, and features like 3.9 and 3.11 are speculative design by standards committee, which is the exact opposite of how these things should work. It's the ultimate unfunded mandate: "hooray, I'm writing a standard, why don't I shove some random crap in there that no-one implements now in existing libraries in the hope I get to set directions". The standard should be minimal and well-defined. This is neither. It's also preposterously self-important. Yeah, I'm so fascinated by the history of the Unicode standard I'm going to start introducing an "age" property so I can regex-match on when characters were entered. There's a certain attitude whereby these guys are just blithely shoveling junk into a standard that no-one needs. If you're desperate for all these exotic character properties put them in a godamn file as a character class and blat them in wherever you need them (yes, it would be nice if regex had macros). Somewhere in all this nonsense is a good standard struggling to get out. They need to shit or get off the pot. Anything that they don't know how to do or that no-one knows how to do should be completely deleted. Anything that they offer 15 different choices as to how to do ... they need to figure out which is the best one and pick that. It's a standards documents, it's supposed to make choices, it's not a potted plant.