5 ms·

> Sure, the complete regex is complex, but it is defined and is hardly unconquerable. If it is a regular expression, then it is not able to match all valid ema

by wtbob 10y ago

> Sure, the complete regex is complex, but it is defined and is hardly unconquerable. If it is a regular expression, then it is not able to match all valid email addresses, because the grammar of email addresses is context-free, and regular expressions can only match regular grammars. It doesn't matter if it is defined or not: if it's a true regular expression, then it simply cannot validate email addresses. (it may, of course, be a context-free expression masquerading as a regular expression) I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest) I suspect that it will, but of course I could be wrong.

Programmatic 10y ago
You are correct: >>> from django.core.validators import EmailValidator >>> EmailValidator()(""""()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)""") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/django/core/validators.py", line 203, in __call__ raise ValidationError(self.message, code=self.code) django.core.exceptions.ValidationError
- wtbob 10y ago
  Well shoot, that's my email address! /s Seriously, though, this is why regexp-based validation is Not Even Wrong. It's like showing up to a gunfight with a keen eye for detail.
tedmiston 10y ago
> ...@(valid)example.org(honest) Are those parens really permitted in the host name? I'm looking at RFC 3696. > 3. Restrictions on email addresses > .... The syntax of the domain part corresponds to that in the previous section. and > 2. Restrictions on domain (DNS) names > Any characters, or combination of bits (as octets), are permitted in DNS names. However, there is a preferred form that is required by most applications. This preferred form has been the only one permitted in the names of top-level domains, or TLDs. .... The LDH rule, as updated, provides that the labels (words or strings separated by periods) that make up a domain name must consist of only the ASCII [ASCII] alphabetic and numeric characters, plus the hyphen. No other symbols or punctuation characters are permitted, nor is blank space. https://tools.ietf.org/html/rfc3696#section-2 https://tools.ietf.org/html/rfc3696#section-2
- wtbob 10y ago
  > > ...@(valid)example.org(honest) > Are those parens really permitted in the host name? I'm looking at RFC 3696. RFC5322 states that an addr-spec consists of a local-part, followed by @, followed by a domain. It states that a domain may be a domain-literal; a domain-literal may begin and end with commented folding whitespace (CFWS). It's all in https://tools.ietf.org/html/rfc5322#section-3.4.1 https://tools.ietf.org/html/rfc5322#section-3.4.1 It's only the dtext portion of the domain-literal which must consist of printable ASCII characters, not including [, ] or \. Yes, foo@bar!baz!quux is a valid RFC5322 email address (decimal 33, i.e. !) is allowed, per the dtext production of the spec) — and so is foo@(comment (nested comment (escaped \(comment)))bar!baz!quux(another (comment (to prove) the point)) Whether one should ever actually use such an address is, of course, another matter entirely.
  - tedmiston 10y ago
    Thank you for that; this has been eyeopening. The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.
    - wtbob 10y ago
      > The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting. It is! Conceivably, it could be used to implement a new kind of domain-less mail system, e.g. foo@$megamail or something.
deathanatos 10y ago
> the grammar of email addresses is context-free I don't think you're really correct about "email addresses" being context-free, or at least, citation, please? When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`. I don't think most people are going to think they can enter that, nor would most programmers even think to implement it. And I certainly wouldn't want to try explaining it to a PM… If you accept that assumption, what about `addr-spec` isn't regular? Also, using that assumption, your "perfectly valud email addresses such as …" would appear to not be valid, as it has unbalanced quotes. (In fact, even under the grammar of `address`, I'm not sure it's valid; it feels like it should be invalid for the same reason, but I've not rigorously checked this.) [1]: https://tools.ietf.org/html/rfc5322#section-3.4.1 https://tools.ietf.org/html/rfc5322#section-3.4.1
- dottrap 10y ago
  I'm not sure if they are context-free, but talks about Parseable Expression Grammars, specifically Lua's LPeg, implies they might be. Conceptually, PEGs are to Context Free Grammars as RegEx is to Regular Expressions. You can find multiple short email validation snippets using Lua LPeg pretty easily, but this is from the Lua creator's talk about LPeg which includes a part about proper RFC822 validation and how complicated it is for regex, but can be concisely done with PEGs. http://program-transformation.org/pub/WGLD/Austin2012/Roberto.pdf http://program-transformation.org/pub/WGLD/Austin2012/Robert... Here is the entire PEG slide. (Looks CFG-ish to me.) address <- mailbox / group group <- phrase ":" mailboxes? ";" phrase <- word ("," word?)* mailboxes <- mailbox ("," mailbox?)* mailbox <- addr_spec / phrase route_addr route_addr <- "<" route? addr_spec ">" route <- ("@" domain) ("," ("@" domain)?)* ":" addr_spec <- local_part "@" domain local_part <- word ("." word)* domain <- sub_domain ("." sub_domain)* sub_domain <- domain_ref / domain_literal domain_ref <- atom domain_literal <- "[" ([^][] / "\" .)* "]" word <- atom / quoted_string atom <- [^] %c()<>@,;:\".[]+ quoted_string <- '"' ([^"\%nl] / "\" .)* '"'
- wtbob 10y ago
thedufer 10y ago
It is true that regular expressions in the CS sense can't parse context-free grammars. However, PCRE, which is what most programmers are talking about when they say "regex", can do so. So you're both kinda right, I guess. But you're being a bit pedantic.
- eyelidlessness 10y ago
  > PCRE, which is what most programmers are talking about I wish that were the case. http://www.regular-expressions.info/refunicode.html http://www.regular-expressions.info/refunicode.html (They used to have a much more useful and concise comparison table but I can't for the life of me find it.)
  - thedufer 10y ago
    Interesting; I've never had to think about how regex interact with Unicode. I guess what I really meant is that programmers are talking about PCRE in terms of power, not in terms of exact syntax. In particular, they have recursive patterns, which are sufficient to pull them up to context-free grammars.
- wtbob 10y ago
  > But you're being a bit pedantic. Technically-correct is the best kind of correct:-) But I do think it's important to note, since there really are differences between regular and 'regular' expressions.
- tedmiston 10y ago
  It definitely can get pedantic in a topic like this, but when all of the interesting bits are in the nuances, I don't mind so much.
ubernostrum 10y ago
I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest) If it doesn't, remind me to patch it to not accept that. I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.
- wtbob 10y ago
  > If it doesn't, remind me to patch it to not accept that. You'd be wrong to do so. The whole point of RFCs and Standards is to take things out of the realm of personal preference. Also, I suspect that a validator which allowed reasonable addresses like jim(somesite)@foo.invalid (which is both a good use of comments and what the + hack emulates) or "Ted Smith"@bar.invalid or "work@home"@jobs.invalid or "William \"Bill\" Jones"@baz.invalid but disallowed unreasonable ones would also be able to solve the halting problem. How many perfectly reasonable allowed features must one use before it's considered abuse? > I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site. An email address is, simply, an address which can be given to an Internet mail server in order for it to properly route an email. All of the addresses I gave are perfectly valid ways to do so. An addresses is a property of the addressee, not of the addressor. In other words: it may be your site, but it's my address.
  - ubernostrum 10y ago
    There are already things that are technically allowed by RFCs for various types of input but should be disallowed because they cause problems. So something being in an RFC doesn't create any kind of binding requirement, and very often the things that get dropped are things that, on contact with the real world, turned out to be bad ideas. Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that. Also, no halting-problem issues at all; my preferred approach would be to disallow certain classes of characters.
Senji 10y ago
1. Strip non alphanumerics except + 2. Remove everything between + and @ 3. Do validation

by wtbob 10y ago

Programmatic 10y ago

You are correct: >>> from django.core.validators import EmailValidator >>> EmailValidator()(""""()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)""") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/django/core/validators.py", line 203, in __call__ raise ValidationError(self.message, code=self.code) django.core.exceptions.ValidationError

wtbob 10y ago
Well shoot, that's my email address! /s Seriously, though, this is why regexp-based validation is Not Even Wrong. It's like showing up to a gunfight with a keen eye for detail.

tedmiston 10y ago

> ...@(valid)example.org(honest) Are those parens really permitted in the host name? I'm looking at RFC 3696. > 3. Restrictions on email addresses > .... The syntax of the domain part corresponds to that in the previous section. and > 2. Restrictions on domain (DNS) names > Any characters, or combination of bits (as octets), are permitted in DNS names. However, there is a preferred form that is required by most applications. This preferred form has been the only one permitted in the names of top-level domains, or TLDs. .... The LDH rule, as updated, provides that the labels (words or strings separated by periods) that make up a domain name must consist of only the ASCII [ASCII] alphabetic and numeric characters, plus the hyphen. No other symbols or punctuation characters are permitted, nor is blank space. https://tools.ietf.org/html/rfc3696#section-2 https://tools.ietf.org/html/rfc3696#section-2

wtbob 10y ago
> > ...@(valid)example.org(honest) > Are those parens really permitted in the host name? I'm looking at RFC 3696. RFC5322 states that an addr-spec consists of a local-part, followed by @, followed by a domain. It states that a domain may be a domain-literal; a domain-literal may begin and end with commented folding whitespace (CFWS). It's all in https://tools.ietf.org/html/rfc5322#section-3.4.1 https://tools.ietf.org/html/rfc5322#section-3.4.1 It's only the dtext portion of the domain-literal which must consist of printable ASCII characters, not including [, ] or \. Yes, foo@bar!baz!quux is a valid RFC5322 email address (decimal 33, i.e. !) is allowed, per the dtext production of the spec) — and so is foo@(comment (nested comment (escaped \(comment)))bar!baz!quux(another (comment (to prove) the point)) Whether one should ever actually use such an address is, of course, another matter entirely.
- tedmiston 10y ago
  Thank you for that; this has been eyeopening. The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.
  - wtbob 10y ago
    > The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting. It is! Conceivably, it could be used to implement a new kind of domain-less mail system, e.g. foo@$megamail or something.

deathanatos 10y ago

> the grammar of email addresses is context-free I don't think you're really correct about "email addresses" being context-free, or at least, citation, please? When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`. I don't think most people are going to think they can enter that, nor would most programmers even think to implement it. And I certainly wouldn't want to try explaining it to a PM… If you accept that assumption, what about `addr-spec` isn't regular? Also, using that assumption, your "perfectly valud email addresses such as …" would appear to not be valid, as it has unbalanced quotes. (In fact, even under the grammar of `address`, I'm not sure it's valid; it feels like it should be invalid for the same reason, but I've not rigorously checked this.) [1]: https://tools.ietf.org/html/rfc5322#section-3.4.1 https://tools.ietf.org/html/rfc5322#section-3.4.1

dottrap 10y ago
I'm not sure if they are context-free, but talks about Parseable Expression Grammars, specifically Lua's LPeg, implies they might be. Conceptually, PEGs are to Context Free Grammars as RegEx is to Regular Expressions. You can find multiple short email validation snippets using Lua LPeg pretty easily, but this is from the Lua creator's talk about LPeg which includes a part about proper RFC822 validation and how complicated it is for regex, but can be concisely done with PEGs. http://program-transformation.org/pub/WGLD/Austin2012/Roberto.pdf http://program-transformation.org/pub/WGLD/Austin2012/Robert... Here is the entire PEG slide. (Looks CFG-ish to me.) address <- mailbox / group group <- phrase ":" mailboxes? ";" phrase <- word ("," word?)* mailboxes <- mailbox ("," mailbox?)* mailbox <- addr_spec / phrase route_addr route_addr <- "<" route? addr_spec ">" route <- ("@" domain) ("," ("@" domain)?)* ":" addr_spec <- local_part "@" domain local_part <- word ("." word)* domain <- sub_domain ("." sub_domain)* sub_domain <- domain_ref / domain_literal domain_ref <- atom domain_literal <- "[" ([^][] / "\" .)* "]" word <- atom / quoted_string atom <- [^] %c()<>@,;:\".[]+ quoted_string <- '"' ([^"\%nl] / "\" .)* '"'
wtbob 10y ago

thedufer 10y ago

It is true that regular expressions in the CS sense can't parse context-free grammars. However, PCRE, which is what most programmers are talking about when they say "regex", can do so. So you're both kinda right, I guess. But you're being a bit pedantic.

eyelidlessness 10y ago
> PCRE, which is what most programmers are talking about I wish that were the case. http://www.regular-expressions.info/refunicode.html http://www.regular-expressions.info/refunicode.html (They used to have a much more useful and concise comparison table but I can't for the life of me find it.)
- thedufer 10y ago
  Interesting; I've never had to think about how regex interact with Unicode. I guess what I really meant is that programmers are talking about PCRE in terms of power, not in terms of exact syntax. In particular, they have recursive patterns, which are sufficient to pull them up to context-free grammars.
wtbob 10y ago
> But you're being a bit pedantic. Technically-correct is the best kind of correct:-) But I do think it's important to note, since there really are differences between regular and 'regular' expressions.
tedmiston 10y ago
It definitely can get pedantic in a topic like this, but when all of the interesting bits are in the nuances, I don't mind so much.

ubernostrum 10y ago

I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest) If it doesn't, remind me to patch it to not accept that. I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.

wtbob 10y ago
> If it doesn't, remind me to patch it to not accept that. You'd be wrong to do so. The whole point of RFCs and Standards is to take things out of the realm of personal preference. Also, I suspect that a validator which allowed reasonable addresses like jim(somesite)@foo.invalid (which is both a good use of comments and what the + hack emulates) or "Ted Smith"@bar.invalid or "work@home"@jobs.invalid or "William \"Bill\" Jones"@baz.invalid but disallowed unreasonable ones would also be able to solve the halting problem. How many perfectly reasonable allowed features must one use before it's considered abuse? > I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site. An email address is, simply, an address which can be given to an Internet mail server in order for it to properly route an email. All of the addresses I gave are perfectly valid ways to do so. An addresses is a property of the addressee, not of the addressor. In other words: it may be your site, but it's my address.
- ubernostrum 10y ago
  There are already things that are technically allowed by RFCs for various types of input but should be disallowed because they cause problems. So something being in an RFC doesn't create any kind of binding requirement, and very often the things that get dropped are things that, on contact with the real world, turned out to be bad ideas. Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that. Also, no halting-problem issues at all; my preferred approach would be to disallow certain classes of characters.

Senji 10y ago

1. Strip non alphanumerics except + 2. Remove everything between + and @ 3. Do validation