5 ms·

Unicode Regular Expressions

96 points by kawera 8y ago

paulpauper 8y ago
Is there a limit to how complicated and ornate a character can be, besides pixels. It's amazing Chinese characters can be rendered so well across many browsers and websites
- squiggleblaz 8y ago
  It depends on the font format. If TrueType doesn't let you code something, you could use an SVG font etc. etc. Almost all fonts in the wild are outline/scalable, so that imposes some limitations.
gok 8y ago
Nice, two pieces of technology which are always implemented incorrectly combined into a single standard.
ktpsns 8y ago
This will make a difference how we write regexes. Look at the level 1 properties: [\p{letter} \p{decimal number}] And compare for instance to the POSIX regexp character classes (a reference: https://en.wikibooks.org/wiki/Regular_Expressions/POSIX_Basic_Regular_Expressions#Character_classes https://en.wikibooks.org/wiki/Regular_Expressions/POSIX_Basi...) [[:alpha:] [:digit:]] Given that the Unicode standard goes much beyond, proper implementations should help to write regexps for the modern web, encountering problems we are typically faced with. It's a pity the standard goes not into detail there, thought (for instance, similiarity of characters).
- JadeNB 8y ago
  > This will make a difference how we write regexes. Look at the level 1 properties: > [\p{letter} \p{decimal number}] > And compare for instance to the POSIX regexp character classes > [[:alpha:] [:digit:]] No need for the future tense; Perl's had that at least since v5.12: http://perldoc.perl.org/5.12.0/perldelta.html#Unicode-overhaul http://perldoc.perl.org/5.12.0/perldelta.html#Unicode-overha... .
rspeer 8y ago
If you want to work with these in Python, use the "regex" package. [1] Not to be confused with the "re" module in the standard library. Also, maybe there should be a (2016) or even a (2004) in the title! (2016 is when this document was last revised, and 2004 is when it became a Unicode standard.) [1] https://pypi.org/project/regex/ https://pypi.org/project/regex/
- xenomachina 8y ago
  Wasn't "regex" also the name of one of re's predecessors?
  - jwilk 8y ago
    Yes. It was deprecated in 1.5(!) and finally removed in 2.5: https://docs.python.org/release/1.5.1/lib/module-regex.html https://docs.python.org/release/1.5.1/lib/module-regex.html https://docs.python.org/2/whatsnew/2.5.html#new-improved-and-removed-modules https://docs.python.org/2/whatsnew/2.5.html#new-improved-and...
- jwilk 8y ago
  I don't think the regex package implements this. At least the hex notation is not supported: >>> import regex >>> regex.compile(r'\u{3040}') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3/dist-packages/regex.py", line 345, in compile return _compile(pattern, flags, kwargs) File "/usr/lib/python3/dist-packages/regex.py", line 507, in _compile caught_exception.pos) _regex_core.error: incomplete escape \u at position 2
  - rspeer 8y ago
    Hmm, okay. It seems to just use Python's unicode escape syntax, instead of what the standard says. But the package does support things like the complex character classes, and identifying word boundaries. (The algorithm for identifying word boundaries in most languages without ASCII/English assumptions is quite complex and useful, as I can say from having tried to reinvent half of it before learning about the standard and the regex package. It's not a panacea -- it won't do anything useful with languages where word boundaries require lexical knowledge, like Chinese, Japanese, and Thai -- but other than that it handles all the edge cases you never would have thought of.)
glangdale 8y ago
This is not really a standard; nearly any behavior which is complicated is left undefined or expressed with vague, aspirational language ("an implementation should do this somehow, I dunno"). It doesn't look like regex implementers were consulted at any time.
- Someone 8y ago
  “Should” likely should be interpreted as defined in https://www.ietf.org/rfc/rfc2119.txt https://www.ietf.org/rfc/rfc2119.txt. I think there are a lot of should’s because the Unicode consortium realizes that there will be variability in feature sets of regular expression libraries. http://unicode.org/reports/tr18/#Conformance http://unicode.org/reports/tr18/#Conformance has quite a few shall’s to counteract that, but still leaves implementations to pick and choose features, as long as they document that.
  - jwilk 8y ago
    HTML rendering of said RFC: https://tools.ietf.org/html/rfc2119 https://tools.ietf.org/html/rfc2119
- garmaine 8y ago
  “Should” has a very precise technical meaning in standards. Mentally it’s best to replace it with “MUST” (they are equivalent).
  - quicklyfrozen 8y ago
    "Must" is equivalent to required, "should" is equivalent to recommended, so not at all the same.
    - garmaine 8y ago
      True enough. I was conflating “should” with “shall” which is definitionally equal to “must”. The edit window has elapsed or I’d correct it.
  - 8y ago
lucio 8y ago
Hearing "Unicode Regular Expressions" makes me think of some other "fantastic" technology combinations: - XML configuration for Windows DLL dependency - CORBA aware DST Library
- speps 8y ago
  > - XML configuration for Windows DLL dependency Oh boy! You're in luck today, you'll learn about side-by-side assemblies and application manifests: https://msdn.microsoft.com/en-us/library/windows/desktop/aa375144%28v=vs.85%29.aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

96 points by kawera 8y ago

Is there a limit to how complicated and ornate a character can be, besides pixels. It's amazing Chinese characters can be rendered so well across many browsers and websites

squiggleblaz 8y ago
It depends on the font format. If TrueType doesn't let you code something, you could use an SVG font etc. etc. Almost all fonts in the wild are outline/scalable, so that imposes some limitations.

gok 8y ago

Nice, two pieces of technology which are always implemented incorrectly combined into a single standard.

ktpsns 8y ago

This will make a difference how we write regexes. Look at the level 1 properties: [\p{letter} \p{decimal number}] And compare for instance to the POSIX regexp character classes (a reference: https://en.wikibooks.org/wiki/Regular_Expressions/POSIX_Basic_Regular_Expressions#Character_classes https://en.wikibooks.org/wiki/Regular_Expressions/POSIX_Basi...) [[:alpha:] [:digit:]] Given that the Unicode standard goes much beyond, proper implementations should help to write regexps for the modern web, encountering problems we are typically faced with. It's a pity the standard goes not into detail there, thought (for instance, similiarity of characters).

JadeNB 8y ago
> This will make a difference how we write regexes. Look at the level 1 properties: > [\p{letter} \p{decimal number}] > And compare for instance to the POSIX regexp character classes > [[:alpha:] [:digit:]] No need for the future tense; Perl's had that at least since v5.12: http://perldoc.perl.org/5.12.0/perldelta.html#Unicode-overhaul http://perldoc.perl.org/5.12.0/perldelta.html#Unicode-overha... .

rspeer 8y ago

If you want to work with these in Python, use the "regex" package. [1] Not to be confused with the "re" module in the standard library. Also, maybe there should be a (2016) or even a (2004) in the title! (2016 is when this document was last revised, and 2004 is when it became a Unicode standard.) [1] https://pypi.org/project/regex/ https://pypi.org/project/regex/

xenomachina 8y ago
Wasn't "regex" also the name of one of re's predecessors?
- jwilk 8y ago
  Yes. It was deprecated in 1.5(!) and finally removed in 2.5: https://docs.python.org/release/1.5.1/lib/module-regex.html https://docs.python.org/release/1.5.1/lib/module-regex.html https://docs.python.org/2/whatsnew/2.5.html#new-improved-and-removed-modules https://docs.python.org/2/whatsnew/2.5.html#new-improved-and...
jwilk 8y ago
I don't think the regex package implements this. At least the hex notation is not supported: >>> import regex >>> regex.compile(r'\u{3040}') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3/dist-packages/regex.py", line 345, in compile return _compile(pattern, flags, kwargs) File "/usr/lib/python3/dist-packages/regex.py", line 507, in _compile caught_exception.pos) _regex_core.error: incomplete escape \u at position 2
- rspeer 8y ago
  Hmm, okay. It seems to just use Python's unicode escape syntax, instead of what the standard says. But the package does support things like the complex character classes, and identifying word boundaries. (The algorithm for identifying word boundaries in most languages without ASCII/English assumptions is quite complex and useful, as I can say from having tried to reinvent half of it before learning about the standard and the regex package. It's not a panacea -- it won't do anything useful with languages where word boundaries require lexical knowledge, like Chinese, Japanese, and Thai -- but other than that it handles all the edge cases you never would have thought of.)

glangdale 8y ago

This is not really a standard; nearly any behavior which is complicated is left undefined or expressed with vague, aspirational language ("an implementation should do this somehow, I dunno"). It doesn't look like regex implementers were consulted at any time.

Someone 8y ago
“Should” likely should be interpreted as defined in https://www.ietf.org/rfc/rfc2119.txt https://www.ietf.org/rfc/rfc2119.txt. I think there are a lot of should’s because the Unicode consortium realizes that there will be variability in feature sets of regular expression libraries. http://unicode.org/reports/tr18/#Conformance http://unicode.org/reports/tr18/#Conformance has quite a few shall’s to counteract that, but still leaves implementations to pick and choose features, as long as they document that.
- jwilk 8y ago
  HTML rendering of said RFC: https://tools.ietf.org/html/rfc2119 https://tools.ietf.org/html/rfc2119
garmaine 8y ago
“Should” has a very precise technical meaning in standards. Mentally it’s best to replace it with “MUST” (they are equivalent).
- quicklyfrozen 8y ago
  "Must" is equivalent to required, "should" is equivalent to recommended, so not at all the same.
  - garmaine 8y ago
    True enough. I was conflating “should” with “shall” which is definitionally equal to “must”. The edit window has elapsed or I’d correct it.
- 8y ago

lucio 8y ago

Hearing "Unicode Regular Expressions" makes me think of some other "fantastic" technology combinations: - XML configuration for Windows DLL dependency - CORBA aware DST Library

speps 8y ago
> - XML configuration for Windows DLL dependency Oh boy! You're in luck today, you'll learn about side-by-side assemblies and application manifests: https://msdn.microsoft.com/en-us/library/windows/desktop/aa375144%28v=vs.85%29.aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...