5 ms·

Hmm, sorry but I don't buy that the "correct way to validate" is not to validate the input. Email addresses aren't a special enough case to be handled differen

Hmm, sorry but I don't buy that the "correct way to validate" is not to validate the input. Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else. Sure, the complete regex is complex, but it is defined and is hardly unconquerable. Look at Django's `EmailValidator` implementation for example [0] that is mature and well tested [1]. The author has not convinced me that ignoring validation is the right choice when options with a scope so thorough exist. [0]: https://github.com/django/django/blob/master/django/core/validators.py#L170 https://github.com/django/django/blob/master/django/core/val... [1]: https://github.com/django/django/blob/a9215b7c36bff232bcc9416309726290dc74a9e8/tests/validators/tests.py#L48 https://github.com/django/django/blob/a9215b7c36bff232bcc941...

acdha 10y ago
You can use a regex as a simple pre-check but you absolutely have to do more than that if you expect high-quality results. Back in the 90s, we ran the customer rewards program mailing list for a mainstream business you've heard of. A [gnarly] regex took care of the gross failures but we still had double-digit percentage of invalid addresses and many spam reports because people mistyped their username, used their old ISP's email address which had been disconnected, etc. The only approach which produced satisfactory results back then was to have a bit of (horrible) PHP code which did a full SMTP connection to deliver the welcome email. Even that wasn't enough to ensure you'd delivered it to the right person, however, so we had to track clicks on an activation link. I would be surprised if the average internet user has gotten significantly more reliable over the last couple of decades and that's born out by the wide variety of misdirected legitimate email my shortname@gmail.com account receives.
- tedmiston 10y ago
  Definitely, I think email validation links are important too. However, it's pretty senseless to let an obviously invalid email address pass all the way through to that layer (and potentially get billed for sending messages to invalid email addresses).
  - zAy0LfpBZLC8mAC 10y ago
    1. There are no "obviously invalid email address[es]". 2. Getting billed for sending emails? WTF?
    - random_rr 10y ago
      You haven't worked in email, I presume. Yes, it does cost money to send emails at a large scale.
      - zAy0LfpBZLC8mAC 10y ago
        I am not sure what you are trying to say. Yes, of course, it costs money. Just like serving websites costs money. And just as checking the syntax of an email address costs money (CPUs cost money and need power to run!). Now, how is that an argument for checking the syntax of an email address to avoid sending a single email and seeing whether it bounces?
        scaryclam 10y ago
        Sending a lot of bounced emails is going to get your mailservers reputation tanked, which is not an acceptable tradeoff for removing email validation. Email validation is actually really important if you send even a medium amount of email. Testing by bouncing is not a very good idea, especially since there are a lot of ways to test the deliverability of the email before you try and send it. That's not to say use a regex instead, you can do some simple checks on the domain etc or you can use one of the many services to check for you (for of a fee of course). It's understandable not to know this unless you've run a medium - large site, so here's some best practices to follow: https://documentation.mailgun.com/best_practices.html#email-best-practices https://documentation.mailgun.com/best_practices.html#email-...
        zAy0LfpBZLC8mAC 10y ago
        > Sending a lot of bounced emails is going to get your mailservers reputation tanked, which is not an acceptable tradeoff for removing email validation. ?!? OK, let's break this down: What could be reasons for an email to be bounced due to an invalid address? 1. Because the address is syntactically invalid: Your own email server will notice and reject/bounce the email. Noone outside your own server will notice. 2. Because the domain doesn't exist or has no inbound mailserver: Your own email server will do the DNS lookup(s) and bounce the email. Noone outside your own server will make any note of that (and you can't find out without doing DNS lookups anyway). 3. Because the localpart doesn't exist: You can't find that out without trying to actually send an email (and consequently risking a bounce). So, what exactly was the point of email validation "without bounces" again? The only thing you can actually do to avoid unnecessary bounces is to make sure your emails have a valid return path and to make sure that you somehow process all the bounces that arrive there to make sure that you don't continue sending emails to addresses that start sending bounces (i.e., that don't exist (any more)). If you send lots of mails, it's probably best to employ VERP for that.
wtbob 10y ago
> Sure, the complete regex is complex, but it is defined and is hardly unconquerable. If it is a regular expression, then it is not able to match all valid email addresses, because the grammar of email addresses is context-free, and regular expressions can only match regular grammars. It doesn't matter if it is defined or not: if it's a true regular expression, then it simply cannot validate email addresses. (it may, of course, be a context-free expression masquerading as a regular expression) I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest) I suspect that it will, but of course I could be wrong.
- Programmatic 10y ago
  You are correct: >>> from django.core.validators import EmailValidator >>> EmailValidator()(""""()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)""") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/django/core/validators.py", line 203, in __call__ raise ValidationError(self.message, code=self.code) django.core.exceptions.ValidationError
  - wtbob 10y ago
    Well shoot, that's my email address! /s Seriously, though, this is why regexp-based validation is Not Even Wrong. It's like showing up to a gunfight with a keen eye for detail.
- tedmiston 10y ago
  > ...@(valid)example.org(honest) Are those parens really permitted in the host name? I'm looking at RFC 3696. > 3. Restrictions on email addresses > .... The syntax of the domain part corresponds to that in the previous section. and > 2. Restrictions on domain (DNS) names > Any characters, or combination of bits (as octets), are permitted in DNS names. However, there is a preferred form that is required by most applications. This preferred form has been the only one permitted in the names of top-level domains, or TLDs. .... The LDH rule, as updated, provides that the labels (words or strings separated by periods) that make up a domain name must consist of only the ASCII [ASCII] alphabetic and numeric characters, plus the hyphen. No other symbols or punctuation characters are permitted, nor is blank space. https://tools.ietf.org/html/rfc3696#section-2 https://tools.ietf.org/html/rfc3696#section-2
honkhonkpants 10y ago
Regarding that testdata, what is the reason for rejecting bare IP literals on the right hand side? It seems pedantic to require the square braces.
- tedmiston 10y ago
  That part is confusing to me. I ran a fresh install of Django to confirm and it behaves as you suggest. Edit: Perhaps only for low digits. Hmm. >>> from django.core.validators import EmailValidator >>> e = EmailValidator() >>> e.__call__('foo@[1.2.3.4]') >>> e.__call__('foo@1.2.3.4') ValidationError: [u'Enter a valid email address.'] >>> e.__call__('foo@0.0.0.0') ValidationError: [u'Enter a valid email address.'] >>> e.__call__('foo@[0.0.0.0]') >>> e.__call__('foo@10.10.10.10') >>> e.__call__('foo@[10.10.10.10]') >>> e.__call__('foo@255.255.255.255') >>> e.__call__('foo@[255.255.255.255]')
  - honkhonkpants 10y ago
    Huh. "Well tested" indeed.
- Senji 10y ago
  You shouldn't be accepting raw ips for e-mail anyway.
zAy0LfpBZLC8mAC 10y ago
> Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else. Yes, they aren't special, and you shouldn't validate any of the other stuff either, unless you actually need to understand the semantics. So much breakage happens because people think that they need to validate all kinds of stuff, it's not funny. People have their legal real names rejected because they're "not a valid name", people have their correct postal address rejected because it's "not a valid postal address" ... just forget it, if someone tells you their name, just believe them it's their name, if someone tells you their postal address, just believe them it's their postal address, if someone tells you their email address ... Oh, and don't ever think about "sanitizing" stuff. Just don't. If you think you need it, you are doing something wrong. The solution to SQL injection is not to disallow people using "'" characters in messages, the solution to cross site scripting is not to prevent people from using "<" characters in comments, ... that's what encoding/escaping is for.
busterarm 10y ago
Right. Wholeheartedly agree. Sanitize all input. If you _KNOW_ input you're receiving is invalid, throw it away. Also whether you use RegExp on email or not totally depends on what you're doing with the address. Are you throwing that email into Salesforce? Great! Salesforce does RegExp validation on all email addresses and the field is required. You _HAVE_ to do it. Solution? Use the same validation that Salesforce does. That's what I do. Also the OPs math about how common invalid email addresses are is wrong. Do you know what the most common way of screwing up an email address is? Hitting tilde when you're tabbing away from the field. And that error is in the tenths or hundreds of a percent range (which is still significant). I have a dataset of 5MM email addresses typed in by jaded call center agents to prove it.
eyelidlessness 10y ago
By all means, validate. But err on the permissive side. The odds are pretty great that you're wrong, and you're preventing real email addresses (and almost certainly real users) by doing so. I'd never looked closely at Django's email validation, but after looking at it I'm inclined to stop using it. Example reason: > # max length for domain name labels is 63 characters per RFC 1034 There is a high likelihood that most of your users have shorter domain names, and also a high likelihood that most if not all registrars currently enforce this. But it also used to be highly likely that domains didn't begin with a numeral, and even though it was disallowed, 3m.com existed. Another: neither the tests nor the patterns account for unicode, which is valid for domain names irrespective of the local part, and if not allowed for the local part by existing providers almost certainly will be eventually. Also, a thing I noticed in the Django tests that is just upsetting: - name: `TEST_DATA` - values: `(validator, value, expected)` (a validator of the data is not the data) - unstructured groups of such that should almost certainly be structured
- jerdfelt 10y ago
  The 63 character limit on domain name labels in RFC 1034/1035 isn't arbitrary, it's a limitation in the way DNS packets are encoded. The upper two bits of the label length are used for compression of labels in DNS packets. Without protocol changes, DNS cannot support labels longer than 63 bytes. However, I agree that validating it is unnecessary.

acdha 10y ago

You can use a regex as a simple pre-check but you absolutely have to do more than that if you expect high-quality results. Back in the 90s, we ran the customer rewards program mailing list for a mainstream business you've heard of. A [gnarly] regex took care of the gross failures but we still had double-digit percentage of invalid addresses and many spam reports because people mistyped their username, used their old ISP's email address which had been disconnected, etc. The only approach which produced satisfactory results back then was to have a bit of (horrible) PHP code which did a full SMTP connection to deliver the welcome email. Even that wasn't enough to ensure you'd delivered it to the right person, however, so we had to track clicks on an activation link. I would be surprised if the average internet user has gotten significantly more reliable over the last couple of decades and that's born out by the wide variety of misdirected legitimate email my shortname@gmail.com account receives.

tedmiston 10y ago
Definitely, I think email validation links are important too. However, it's pretty senseless to let an obviously invalid email address pass all the way through to that layer (and potentially get billed for sending messages to invalid email addresses).
- zAy0LfpBZLC8mAC 10y ago
  1. There are no "obviously invalid email address[es]". 2. Getting billed for sending emails? WTF?
  - random_rr 10y ago
    You haven't worked in email, I presume. Yes, it does cost money to send emails at a large scale.
    - zAy0LfpBZLC8mAC 10y ago
      I am not sure what you are trying to say. Yes, of course, it costs money. Just like serving websites costs money. And just as checking the syntax of an email address costs money (CPUs cost money and need power to run!). Now, how is that an argument for checking the syntax of an email address to avoid sending a single email and seeing whether it bounces?
      - scaryclam 10y ago
        Sending a lot of bounced emails is going to get your mailservers reputation tanked, which is not an acceptable tradeoff for removing email validation. Email validation is actually really important if you send even a medium amount of email. Testing by bouncing is not a very good idea, especially since there are a lot of ways to test the deliverability of the email before you try and send it. That's not to say use a regex instead, you can do some simple checks on the domain etc or you can use one of the many services to check for you (for of a fee of course). It's understandable not to know this unless you've run a medium - large site, so here's some best practices to follow: https://documentation.mailgun.com/best_practices.html#email-best-practices https://documentation.mailgun.com/best_practices.html#email-...
        zAy0LfpBZLC8mAC 10y ago
        > Sending a lot of bounced emails is going to get your mailservers reputation tanked, which is not an acceptable tradeoff for removing email validation. ?!? OK, let's break this down: What could be reasons for an email to be bounced due to an invalid address? 1. Because the address is syntactically invalid: Your own email server will notice and reject/bounce the email. Noone outside your own server will notice. 2. Because the domain doesn't exist or has no inbound mailserver: Your own email server will do the DNS lookup(s) and bounce the email. Noone outside your own server will make any note of that (and you can't find out without doing DNS lookups anyway). 3. Because the localpart doesn't exist: You can't find that out without trying to actually send an email (and consequently risking a bounce). So, what exactly was the point of email validation "without bounces" again? The only thing you can actually do to avoid unnecessary bounces is to make sure your emails have a valid return path and to make sure that you somehow process all the bounces that arrive there to make sure that you don't continue sending emails to addresses that start sending bounces (i.e., that don't exist (any more)). If you send lots of mails, it's probably best to employ VERP for that.

wtbob 10y ago

> Sure, the complete regex is complex, but it is defined and is hardly unconquerable. If it is a regular expression, then it is not able to match all valid email addresses, because the grammar of email addresses is context-free, and regular expressions can only match regular grammars. It doesn't matter if it is defined or not: if it's a true regular expression, then it simply cannot validate email addresses. (it may, of course, be a context-free expression masquerading as a regular expression) I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest) I suspect that it will, but of course I could be wrong.

Programmatic 10y ago
You are correct: >>> from django.core.validators import EmailValidator >>> EmailValidator()(""""()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)""") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/django/core/validators.py", line 203, in __call__ raise ValidationError(self.message, code=self.code) django.core.exceptions.ValidationError
- wtbob 10y ago
  Well shoot, that's my email address! /s Seriously, though, this is why regexp-based validation is Not Even Wrong. It's like showing up to a gunfight with a keen eye for detail.
tedmiston 10y ago
> ...@(valid)example.org(honest) Are those parens really permitted in the host name? I'm looking at RFC 3696. > 3. Restrictions on email addresses > .... The syntax of the domain part corresponds to that in the previous section. and > 2. Restrictions on domain (DNS) names > Any characters, or combination of bits (as octets), are permitted in DNS names. However, there is a preferred form that is required by most applications. This preferred form has been the only one permitted in the names of top-level domains, or TLDs. .... The LDH rule, as updated, provides that the labels (words or strings separated by periods) that make up a domain name must consist of only the ASCII [ASCII] alphabetic and numeric characters, plus the hyphen. No other symbols or punctuation characters are permitted, nor is blank space. https://tools.ietf.org/html/rfc3696#section-2 https://tools.ietf.org/html/rfc3696#section-2

honkhonkpants 10y ago

Regarding that testdata, what is the reason for rejecting bare IP literals on the right hand side? It seems pedantic to require the square braces.

tedmiston 10y ago
That part is confusing to me. I ran a fresh install of Django to confirm and it behaves as you suggest. Edit: Perhaps only for low digits. Hmm. >>> from django.core.validators import EmailValidator >>> e = EmailValidator() >>> e.__call__('foo@[1.2.3.4]') >>> e.__call__('foo@1.2.3.4') ValidationError: [u'Enter a valid email address.'] >>> e.__call__('foo@0.0.0.0') ValidationError: [u'Enter a valid email address.'] >>> e.__call__('foo@[0.0.0.0]') >>> e.__call__('foo@10.10.10.10') >>> e.__call__('foo@[10.10.10.10]') >>> e.__call__('foo@255.255.255.255') >>> e.__call__('foo@[255.255.255.255]')
- honkhonkpants 10y ago
  Huh. "Well tested" indeed.
Senji 10y ago
You shouldn't be accepting raw ips for e-mail anyway.

zAy0LfpBZLC8mAC 10y ago

> Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else. Yes, they aren't special, and you shouldn't validate any of the other stuff either, unless you actually need to understand the semantics. So much breakage happens because people think that they need to validate all kinds of stuff, it's not funny. People have their legal real names rejected because they're "not a valid name", people have their correct postal address rejected because it's "not a valid postal address" ... just forget it, if someone tells you their name, just believe them it's their name, if someone tells you their postal address, just believe them it's their postal address, if someone tells you their email address ... Oh, and don't ever think about "sanitizing" stuff. Just don't. If you think you need it, you are doing something wrong. The solution to SQL injection is not to disallow people using "'" characters in messages, the solution to cross site scripting is not to prevent people from using "<" characters in comments, ... that's what encoding/escaping is for.

busterarm 10y ago

Right. Wholeheartedly agree. Sanitize all input. If you _KNOW_ input you're receiving is invalid, throw it away. Also whether you use RegExp on email or not totally depends on what you're doing with the address. Are you throwing that email into Salesforce? Great! Salesforce does RegExp validation on all email addresses and the field is required. You _HAVE_ to do it. Solution? Use the same validation that Salesforce does. That's what I do. Also the OPs math about how common invalid email addresses are is wrong. Do you know what the most common way of screwing up an email address is? Hitting tilde when you're tabbing away from the field. And that error is in the tenths or hundreds of a percent range (which is still significant). I have a dataset of 5MM email addresses typed in by jaded call center agents to prove it.

eyelidlessness 10y ago

By all means, validate. But err on the permissive side. The odds are pretty great that you're wrong, and you're preventing real email addresses (and almost certainly real users) by doing so. I'd never looked closely at Django's email validation, but after looking at it I'm inclined to stop using it. Example reason: > # max length for domain name labels is 63 characters per RFC 1034 There is a high likelihood that most of your users have shorter domain names, and also a high likelihood that most if not all registrars currently enforce this. But it also used to be highly likely that domains didn't begin with a numeral, and even though it was disallowed, 3m.com existed. Another: neither the tests nor the patterns account for unicode, which is valid for domain names irrespective of the local part, and if not allowed for the local part by existing providers almost certainly will be eventually. Also, a thing I noticed in the Django tests that is just upsetting: - name: `TEST_DATA` - values: `(validator, value, expected)` (a validator of the data is not the data) - unstructured groups of such that should almost certainly be structured

jerdfelt 10y ago
The 63 character limit on domain name labels in RFC 1034/1035 isn't arbitrary, it's a limitation in the way DNS packets are encoded. The upper two bits of the label length are used for compression of labels in DNS packets. Without protocol changes, DNS cannot support labels longer than 63 bytes. However, I agree that validating it is unnecessary.