5 ms·

Don't use regular expressions for those things if you can avoid it. See http://www.regular-expressions.info/email.html http://www.regular-expressions.info/emai

Don't use regular expressions for those things if you can avoid it. See http://www.regular-expressions.info/email.html http://www.regular-expressions.info/email.html. My URL (IRI) validation regex is a very complicated work in progress and depends on being post-processed by a Python function to handle Unicode characters outside the BMP. It has hundreds of lines of unit tests (not shown): unreserved = ur'(?:[A-Za-z0-9\-\._~]|%s)' % ucschar pct_encoded= ur'%[0-9A-Fa-f]{2}' sub_delims = ur"[!$&'()*+,;=]" pchar = r'(?:%s|%s|%s|[:@])' % (unreserved, pct_encoded, sub_delims) dec_octet = ur'(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])' ipv4address = ur'%s\.%s\.%s\.%s' % (dec_octet, dec_octet, dec_octet, dec_octet) h16 = ur'(?:[0-9A-Fa-f]{1,4})' # 16 bits of address represented in hexadecima ls32 = ur'(?:%s:%s|%s)' % (h16, h16, ipv4address) # least-significant 32 bits of address ipv6address = ( u'(?:' + ur'(?:h16:){6}ls32' + u'|' + ur'::(?:h16:){5}ls32' + u'|' + ur'(h16)?::(?:h16:){4}ls32' + u'|' + ur'((?:h16:){0,1}h16)?::(?:h16:){3}ls32' + u'|' + ur'((?:h16:){0,2}h16)?::(?:h16:){2}ls32' + u'|' + ur'((?:h16:){0,3}h16)?::(?:h16:){1}ls32' + u'|' + ur'((?:h16:){0,4}h16)?::ls32' + u'|' + ur'((?:h16:){0,5}h16)?::h16' + u'|' + ur'((?:h16:){0,6}h16)?::' + u')' ).replace(u"h16", h16).replace(u"ls32",ls32) ipvfuture = ur'v[0-9A-Fa-f]+\.(?:%s|%s|:)+' % (unreserved, sub_delims) ip_literal = ur'\[(?:%s|%s)\]' % (ipv6address, ipvfuture) scheme = ur'(?:[A-Za-z][A-Za-z0-9+\-\.]*)' userinfo = ur'(?:%s|%s|%s|:)*' % (unreserved, pct_encoded, sub_delims) reg_name = ur'(?:%s|%s|%s)*' % (unreserved, pct_encoded, sub_delims) host = ur'(?:%s|%s|%s)' % (ip_literal, ipv4address, reg_name) port = ur'[0-9]*' authority = ur'(?:(?:%s@)?%s(?:\:%s)?)' % (userinfo, host, port) segment = ur'%s*' % pchar segment_nz = ur'%s+' % pchar path_abempty = ur'(?:/%s)*' % ( segment) path_absolute = ur'(?:/(?:%s(?:/%s)*)?)' % (segment_nz, segment) path_rootless = ur'(?:%s(?:/%s)*)' % (segment_nz, segment) path_empty = ur'' hier_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_rootless, path_empty) private = ur'[\ue000-\uF8FF]' # see replaceSurrogatePairs for handling of non-BMP chars query = ur'(?:%s|[/?]|%s)*' % (pchar, private) fragment = ur'(?:%s|[/?])*' % (pchar) iri = ur'%s:%s(?:\?%s)?(?:#%s)?' % (scheme, hier_part, query, fragment) segment_nz_nc = ur'(?:%s|%s|%s|@)+' % (unreserved, pct_encoded, sub_delims) path_noscheme = ur'(?:%s(/%s)*)' % (segment_nz_nc, segment) relative_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_noscheme, path_empty) relative_ref = ur'(?:%s(?:\?%s)?(?:#%s)?)' % (relative_part, query, fragment) iri_reference = ur'(?:%s|%s)' % (iri, relative_ref)

alexk 18y ago
why don't you use any parser generator (e.g. PLY) and write a grammar instead?
- briansmith 18y ago
  That is what I've done in the past with non-Python projects using ANTLR or other parser generator tools. I haven't done that yet because: * I want to release the software that contains this validator and I want to limit the external build-time and (especially) run-time dependencies. So far, the application requires nothing except Python 2.4. * The RE code is easier to unit test. When you use a parser generator you usually have to nominate start productions in your grammar. In order to unit test you have to add lots of extra start productions. That tends to substantially impact the readability, performance, and/or size of the generated code. * I keep hearing that people are getting by with regular expressions to do complex stuff but traditionally I've always jumped directly to ANTLR and other similar systems. I wanted to try doing this just using regular expressions to get a good grasp of what those tools are buying me and to find out why so many people just end up using regular expressions. I found that, because this tool is just a simple validator/recognizer, those advanced tools don't really buy me much here. As you can see, the Python+RE code I posted directly mirrors the RFC 3987 grammar. If I switched to PLY or something similar, I wouldn't save anything in terms of LoC and readability wouldn't be that much improved. * I've heard many times that it is best to minimize the amount of Python code that executes in the critical paths of a program; instead, we will get better performance by using native libraries (like the re module). Sometime I will probably try out a code generator as an experiment so that I can measure the performance of the re module vs. a bunch of generated Python code.
  - rcoder 18y ago
    > I've heard many times that it is best to minimize the > amount of Python code that executes in the critical > paths of a program; instead, we will get better > performance by using native libraries (like the re > module). Keep in mind that, while the 're' module itself may be written in C, it is in effect an interpreter for another language (regexps). You can easily build a regular expression that will happily backtrack and capture its way into a performance black hole, esp. since you can't trace the individual sub-patterns using normal Python profiling tools. Benchmarking the two implementations, as you suggest, is of course the only way to be sure in your case.

alexk 18y ago

why don't you use any parser generator (e.g. PLY) and write a grammar instead?

briansmith 18y ago
That is what I've done in the past with non-Python projects using ANTLR or other parser generator tools. I haven't done that yet because: * I want to release the software that contains this validator and I want to limit the external build-time and (especially) run-time dependencies. So far, the application requires nothing except Python 2.4. * The RE code is easier to unit test. When you use a parser generator you usually have to nominate start productions in your grammar. In order to unit test you have to add lots of extra start productions. That tends to substantially impact the readability, performance, and/or size of the generated code. * I keep hearing that people are getting by with regular expressions to do complex stuff but traditionally I've always jumped directly to ANTLR and other similar systems. I wanted to try doing this just using regular expressions to get a good grasp of what those tools are buying me and to find out why so many people just end up using regular expressions. I found that, because this tool is just a simple validator/recognizer, those advanced tools don't really buy me much here. As you can see, the Python+RE code I posted directly mirrors the RFC 3987 grammar. If I switched to PLY or something similar, I wouldn't save anything in terms of LoC and readability wouldn't be that much improved. * I've heard many times that it is best to minimize the amount of Python code that executes in the critical paths of a program; instead, we will get better performance by using native libraries (like the re module). Sometime I will probably try out a code generator as an experiment so that I can measure the performance of the re module vs. a bunch of generated Python code.
- rcoder 18y ago
  > I've heard many times that it is best to minimize the > amount of Python code that executes in the critical > paths of a program; instead, we will get better > performance by using native libraries (like the re > module). Keep in mind that, while the 're' module itself may be written in C, it is in effect an interpreter for another language (regexps). You can easily build a regular expression that will happily backtrack and capture its way into a performance black hole, esp. since you can't trace the individual sub-patterns using normal Python profiling tools. Benchmarking the two implementations, as you suggest, is of course the only way to be sure in your case.