4 ms·

meh disappointed. Could any of you guys give me one liners for email and URL validation?

briansmith 18y ago
Don't use regular expressions for those things if you can avoid it. See http://www.regular-expressions.info/email.html http://www.regular-expressions.info/email.html. My URL (IRI) validation regex is a very complicated work in progress and depends on being post-processed by a Python function to handle Unicode characters outside the BMP. It has hundreds of lines of unit tests (not shown): unreserved = ur'(?:[A-Za-z0-9\-\._~]|%s)' % ucschar pct_encoded= ur'%[0-9A-Fa-f]{2}' sub_delims = ur"[!$&'()*+,;=]" pchar = r'(?:%s|%s|%s|[:@])' % (unreserved, pct_encoded, sub_delims) dec_octet = ur'(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])' ipv4address = ur'%s\.%s\.%s\.%s' % (dec_octet, dec_octet, dec_octet, dec_octet) h16 = ur'(?:[0-9A-Fa-f]{1,4})' # 16 bits of address represented in hexadecima ls32 = ur'(?:%s:%s|%s)' % (h16, h16, ipv4address) # least-significant 32 bits of address ipv6address = ( u'(?:' + ur'(?:h16:){6}ls32' + u'|' + ur'::(?:h16:){5}ls32' + u'|' + ur'(h16)?::(?:h16:){4}ls32' + u'|' + ur'((?:h16:){0,1}h16)?::(?:h16:){3}ls32' + u'|' + ur'((?:h16:){0,2}h16)?::(?:h16:){2}ls32' + u'|' + ur'((?:h16:){0,3}h16)?::(?:h16:){1}ls32' + u'|' + ur'((?:h16:){0,4}h16)?::ls32' + u'|' + ur'((?:h16:){0,5}h16)?::h16' + u'|' + ur'((?:h16:){0,6}h16)?::' + u')' ).replace(u"h16", h16).replace(u"ls32",ls32) ipvfuture = ur'v[0-9A-Fa-f]+\.(?:%s|%s|:)+' % (unreserved, sub_delims) ip_literal = ur'\[(?:%s|%s)\]' % (ipv6address, ipvfuture) scheme = ur'(?:[A-Za-z][A-Za-z0-9+\-\.]*)' userinfo = ur'(?:%s|%s|%s|:)*' % (unreserved, pct_encoded, sub_delims) reg_name = ur'(?:%s|%s|%s)*' % (unreserved, pct_encoded, sub_delims) host = ur'(?:%s|%s|%s)' % (ip_literal, ipv4address, reg_name) port = ur'[0-9]*' authority = ur'(?:(?:%s@)?%s(?:\:%s)?)' % (userinfo, host, port) segment = ur'%s*' % pchar segment_nz = ur'%s+' % pchar path_abempty = ur'(?:/%s)*' % ( segment) path_absolute = ur'(?:/(?:%s(?:/%s)*)?)' % (segment_nz, segment) path_rootless = ur'(?:%s(?:/%s)*)' % (segment_nz, segment) path_empty = ur'' hier_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_rootless, path_empty) private = ur'[\ue000-\uF8FF]' # see replaceSurrogatePairs for handling of non-BMP chars query = ur'(?:%s|[/?]|%s)*' % (pchar, private) fragment = ur'(?:%s|[/?])*' % (pchar) iri = ur'%s:%s(?:\?%s)?(?:#%s)?' % (scheme, hier_part, query, fragment) segment_nz_nc = ur'(?:%s|%s|%s|@)+' % (unreserved, pct_encoded, sub_delims) path_noscheme = ur'(?:%s(/%s)*)' % (segment_nz_nc, segment) relative_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_noscheme, path_empty) relative_ref = ur'(?:%s(?:\?%s)?(?:#%s)?)' % (relative_part, query, fragment) iri_reference = ur'(?:%s|%s)' % (iri, relative_ref)
- alexk 18y ago
  why don't you use any parser generator (e.g. PLY) and write a grammar instead?
  - briansmith 18y ago
    That is what I've done in the past with non-Python projects using ANTLR or other parser generator tools. I haven't done that yet because: * I want to release the software that contains this validator and I want to limit the external build-time and (especially) run-time dependencies. So far, the application requires nothing except Python 2.4. * The RE code is easier to unit test. When you use a parser generator you usually have to nominate start productions in your grammar. In order to unit test you have to add lots of extra start productions. That tends to substantially impact the readability, performance, and/or size of the generated code. * I keep hearing that people are getting by with regular expressions to do complex stuff but traditionally I've always jumped directly to ANTLR and other similar systems. I wanted to try doing this just using regular expressions to get a good grasp of what those tools are buying me and to find out why so many people just end up using regular expressions. I found that, because this tool is just a simple validator/recognizer, those advanced tools don't really buy me much here. As you can see, the Python+RE code I posted directly mirrors the RFC 3987 grammar. If I switched to PLY or something similar, I wouldn't save anything in terms of LoC and readability wouldn't be that much improved. * I've heard many times that it is best to minimize the amount of Python code that executes in the critical paths of a program; instead, we will get better performance by using native libraries (like the re module). Sometime I will probably try out a code generator as an experiment so that I can measure the performance of the re module vs. a bunch of generated Python code.
    - rcoder 18y ago
      > I've heard many times that it is best to minimize the > amount of Python code that executes in the critical > paths of a program; instead, we will get better > performance by using native libraries (like the re > module). Keep in mind that, while the 're' module itself may be written in C, it is in effect an interpreter for another language (regexps). You can easily build a regular expression that will happily backtrack and capture its way into a performance black hole, esp. since you can't trace the individual sub-patterns using normal Python profiling tools. Benchmarking the two implementations, as you suggest, is of course the only way to be sure in your case.
deleted 18y ago
[deleted]
deleted 18y ago
[deleted]
hs 18y ago
i use this for email: /[-\.\\w]*@\\w+(\.\\w+)+/
- thwarted 18y ago
  DO NOT USE THIS REGULAR EXPRESSION FOR ANYTHING. This is a terrible example. First off, you're using double-quoted syntax, but it's not in double quotes. Additionally, you're confusing when regular expression metacharacters need to be escape because they're metacharacters and when they need to be escaped because they are in a double-quoted string. The PHP documentation is particularly terrible in this regard (telling you to put regular expressions in double quoted strings rather than single quoted, because PHP doesn't have a regular expression type). [-\.\\w] means match a dash, a dot, a backslash and the w character. Secondly, it's not anchored to the start or end of the input. Thirdly, the LHS can be empty according to this regular expression. Lastly, we've finally uncovered who it is that's keeping everyone from using + on the LHS to do sendmail style +folder references.
  - litewulf 18y ago
    Depending on the regex function you use, you may or may not need the anchoring. FWIW.
  - hs 18y ago
    actually i didn't use PHP, the code i submitted is 'ported' using /regex/ ... i haven't tested it in PHP (i don't have) ... but that what i might use if i were forced to use PHP the actual code is in newlisp: <code> (set 'p1 (regex-comp {[-\.\w]@\w+(\.\w+)+})) ;compile (map (lambda (f) (replace p1 (read-file f) (push $0 E) 0x10000) ) (directory "." "php|htm|html")) (set 'E (unique E)) (save "email.db" 'E) </code> my newlisp code is not used for validation, it's for scrapping emails from ~30000 html pages totalling 870 MB <pre> ~/dl $ find . | wc 29546 29548 1423727 ~/dl $ du -h 870M </pre> I used more complex regexes (slow) but only marginal improvement (< 0.1%) and a lot of noises This code is pretty fast, almost an order magnitude faster than more complex, complete email regex [-\.\\w] means ... match - or \. or word (a-zA-Z0-9) character ad nauseum (can put more chars inside [] but the regex is slower) i didn't use it for validation, so no anchoring needed ... but it's an easy modification anyway, this is the code i use for real world - i won't pretend that it's perfect for every corner cases, but hey it works fast for me :D short code solving 90+% cases
iigs 18y ago
Surprisingly, this is fairly hard to do with regular expressions. You really want to outsource this to someone else's library. But, if you must, here's the regex from one library: http://ex-parrot.com/~pdw/Mail-RFC822-Address.html http://ex-parrot.com/~pdw/Mail-RFC822-Address.html Not exactly a one liner, though.
olavk 18y ago
The value of regex-validating emails and urls are limited. Validation is mostly useful for flagging completely malformed input, like people writing "asdf" or "I dont have an email, plz use fed-ex.". Regex validation will not catch common typos (or made-up emails) because they tend to look like correct emails. So the intricate, syntactically correct email validation regexes are a solution in search of a problem. If you really want to be sure that the email is correct, you have to send a confirmation mail. For your input validation, just check that there is a "@" somewhere.
- Jasber 18y ago
  People also try to look up the MX records for the domain to see if it has a valid mail server. This works 95% of the time--but breaks in certain cases leading to false positives. As olavk said, just do basic validation (@ exists) and send off a confirmation e-mail. It works 100% of the time.

Don't use regular expressions for those things if you can avoid it. See http://www.regular-expressions.info/email.html http://www.regular-expressions.info/email.html. My URL (IRI) validation regex is a very complicated work in progress and depends on being post-processed by a Python function to handle Unicode characters outside the BMP. It has hundreds of lines of unit tests (not shown): unreserved = ur'(?:[A-Za-z0-9\-\._~]|%s)' % ucschar pct_encoded= ur'%[0-9A-Fa-f]{2}' sub_delims = ur"[!$&'()*+,;=]" pchar = r'(?:%s|%s|%s|[:@])' % (unreserved, pct_encoded, sub_delims) dec_octet = ur'(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])' ipv4address = ur'%s\.%s\.%s\.%s' % (dec_octet, dec_octet, dec_octet, dec_octet) h16 = ur'(?:[0-9A-Fa-f]{1,4})' # 16 bits of address represented in hexadecima ls32 = ur'(?:%s:%s|%s)' % (h16, h16, ipv4address) # least-significant 32 bits of address ipv6address = ( u'(?:' + ur'(?:h16:){6}ls32' + u'|' + ur'::(?:h16:){5}ls32' + u'|' + ur'(h16)?::(?:h16:){4}ls32' + u'|' + ur'((?:h16:){0,1}h16)?::(?:h16:){3}ls32' + u'|' + ur'((?:h16:){0,2}h16)?::(?:h16:){2}ls32' + u'|' + ur'((?:h16:){0,3}h16)?::(?:h16:){1}ls32' + u'|' + ur'((?:h16:){0,4}h16)?::ls32' + u'|' + ur'((?:h16:){0,5}h16)?::h16' + u'|' + ur'((?:h16:){0,6}h16)?::' + u')' ).replace(u"h16", h16).replace(u"ls32",ls32) ipvfuture = ur'v[0-9A-Fa-f]+\.(?:%s|%s|:)+' % (unreserved, sub_delims) ip_literal = ur'\[(?:%s|%s)\]' % (ipv6address, ipvfuture) scheme = ur'(?:[A-Za-z][A-Za-z0-9+\-\.]*)' userinfo = ur'(?:%s|%s|%s|:)*' % (unreserved, pct_encoded, sub_delims) reg_name = ur'(?:%s|%s|%s)*' % (unreserved, pct_encoded, sub_delims) host = ur'(?:%s|%s|%s)' % (ip_literal, ipv4address, reg_name) port = ur'[0-9]*' authority = ur'(?:(?:%s@)?%s(?:\:%s)?)' % (userinfo, host, port) segment = ur'%s*' % pchar segment_nz = ur'%s+' % pchar path_abempty = ur'(?:/%s)*' % ( segment) path_absolute = ur'(?:/(?:%s(?:/%s)*)?)' % (segment_nz, segment) path_rootless = ur'(?:%s(?:/%s)*)' % (segment_nz, segment) path_empty = ur'' hier_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_rootless, path_empty) private = ur'[\ue000-\uF8FF]' # see replaceSurrogatePairs for handling of non-BMP chars query = ur'(?:%s|[/?]|%s)*' % (pchar, private) fragment = ur'(?:%s|[/?])*' % (pchar) iri = ur'%s:%s(?:\?%s)?(?:#%s)?' % (scheme, hier_part, query, fragment) segment_nz_nc = ur'(?:%s|%s|%s|@)+' % (unreserved, pct_encoded, sub_delims) path_noscheme = ur'(?:%s(/%s)*)' % (segment_nz_nc, segment) relative_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_noscheme, path_empty) relative_ref = ur'(?:%s(?:\?%s)?(?:#%s)?)' % (relative_part, query, fragment) iri_reference = ur'(?:%s|%s)' % (iri, relative_ref)

alexk 18y ago
why don't you use any parser generator (e.g. PLY) and write a grammar instead?
- briansmith 18y ago
  That is what I've done in the past with non-Python projects using ANTLR or other parser generator tools. I haven't done that yet because: * I want to release the software that contains this validator and I want to limit the external build-time and (especially) run-time dependencies. So far, the application requires nothing except Python 2.4. * The RE code is easier to unit test. When you use a parser generator you usually have to nominate start productions in your grammar. In order to unit test you have to add lots of extra start productions. That tends to substantially impact the readability, performance, and/or size of the generated code. * I keep hearing that people are getting by with regular expressions to do complex stuff but traditionally I've always jumped directly to ANTLR and other similar systems. I wanted to try doing this just using regular expressions to get a good grasp of what those tools are buying me and to find out why so many people just end up using regular expressions. I found that, because this tool is just a simple validator/recognizer, those advanced tools don't really buy me much here. As you can see, the Python+RE code I posted directly mirrors the RFC 3987 grammar. If I switched to PLY or something similar, I wouldn't save anything in terms of LoC and readability wouldn't be that much improved. * I've heard many times that it is best to minimize the amount of Python code that executes in the critical paths of a program; instead, we will get better performance by using native libraries (like the re module). Sometime I will probably try out a code generator as an experiment so that I can measure the performance of the re module vs. a bunch of generated Python code.
  - rcoder 18y ago
    > I've heard many times that it is best to minimize the > amount of Python code that executes in the critical > paths of a program; instead, we will get better > performance by using native libraries (like the re > module). Keep in mind that, while the 're' module itself may be written in C, it is in effect an interpreter for another language (regexps). You can easily build a regular expression that will happily backtrack and capture its way into a performance black hole, esp. since you can't trace the individual sub-patterns using normal Python profiling tools. Benchmarking the two implementations, as you suggest, is of course the only way to be sure in your case.

deleted 18y ago

[deleted]

deleted 18y ago

[deleted]

hs 18y ago

i use this for email: /[-\.\\w]*@\\w+(\.\\w+)+/

thwarted 18y ago
DO NOT USE THIS REGULAR EXPRESSION FOR ANYTHING. This is a terrible example. First off, you're using double-quoted syntax, but it's not in double quotes. Additionally, you're confusing when regular expression metacharacters need to be escape because they're metacharacters and when they need to be escaped because they are in a double-quoted string. The PHP documentation is particularly terrible in this regard (telling you to put regular expressions in double quoted strings rather than single quoted, because PHP doesn't have a regular expression type). [-\.\\w] means match a dash, a dot, a backslash and the w character. Secondly, it's not anchored to the start or end of the input. Thirdly, the LHS can be empty according to this regular expression. Lastly, we've finally uncovered who it is that's keeping everyone from using + on the LHS to do sendmail style +folder references.
- litewulf 18y ago
  Depending on the regex function you use, you may or may not need the anchoring. FWIW.
- hs 18y ago
  actually i didn't use PHP, the code i submitted is 'ported' using /regex/ ... i haven't tested it in PHP (i don't have) ... but that what i might use if i were forced to use PHP the actual code is in newlisp: <code> (set 'p1 (regex-comp {[-\.\w]@\w+(\.\w+)+})) ;compile (map (lambda (f) (replace p1 (read-file f) (push $0 E) 0x10000) ) (directory "." "php|htm|html")) (set 'E (unique E)) (save "email.db" 'E) </code> my newlisp code is not used for validation, it's for scrapping emails from ~30000 html pages totalling 870 MB <pre> ~/dl $ find . | wc 29546 29548 1423727 ~/dl $ du -h 870M </pre> I used more complex regexes (slow) but only marginal improvement (< 0.1%) and a lot of noises This code is pretty fast, almost an order magnitude faster than more complex, complete email regex [-\.\\w] means ... match - or \. or word (a-zA-Z0-9) character ad nauseum (can put more chars inside [] but the regex is slower) i didn't use it for validation, so no anchoring needed ... but it's an easy modification anyway, this is the code i use for real world - i won't pretend that it's perfect for every corner cases, but hey it works fast for me :D short code solving 90+% cases

iigs 18y ago

Surprisingly, this is fairly hard to do with regular expressions. You really want to outsource this to someone else's library. But, if you must, here's the regex from one library: http://ex-parrot.com/~pdw/Mail-RFC822-Address.html http://ex-parrot.com/~pdw/Mail-RFC822-Address.html Not exactly a one liner, though.

olavk 18y ago

The value of regex-validating emails and urls are limited. Validation is mostly useful for flagging completely malformed input, like people writing "asdf" or "I dont have an email, plz use fed-ex.". Regex validation will not catch common typos (or made-up emails) because they tend to look like correct emails. So the intricate, syntactically correct email validation regexes are a solution in search of a problem. If you really want to be sure that the email is correct, you have to send a confirmation mail. For your input validation, just check that there is a "@" somewhere.

Jasber 18y ago
People also try to look up the MX records for the domain to see if it has a valid mail server. This works 95% of the time--but breaks in certain cases leading to false positives. As olavk said, just do basic validation (@ exists) and send off a confirmation e-mail. It works 100% of the time.