6 ms·

Besides the usual regex aches and pains, the grammar for email addresses is far more complex than most people realize. According to a highly-voted Stack Overflo

Besides the usual regex aches and pains, the grammar for email addresses is far more complex than most people realize. According to a highly-voted Stack Overflow answer [1], the current RFC-specified grammar for addresses can't even be matched with regex alone. Combining the edge cases of the grammar with (say) Unicode normalization sounds like a recipe for hours of fun. [1] https://stackoverflow.com/questions/201323/using-a-regular-expression-to-validate-an-email-address https://stackoverflow.com/questions/201323/using-a-regular-e...

kccqzy 11y ago
I find that quite unbelievable. When I had a similar problem last year, the first resource I found was a W3C specification[1] about <input type=email>. The specification clearly states that email addresses should match: /^[a-zA-Z0-9.!#$%&’*+/=?^_`{|}~-]+@[a-zA-Z0-9-]+(?:\.[a-zA-Z0-9-]+)*$/ Since this is an official W3C doc, I see no reason why people shouldn't use this. Edit: There is also a version by WHATWG[2] here: /^[a-zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/ It apparently does a more thorough validation than the W3C one in the domain part, but the difference between the two is not apparent in practice. [1]: http://www.w3.org/TR/html-markup/input.email.html http://www.w3.org/TR/html-markup/input.email.html [2]: https://html.spec.whatwg.org/multipage/forms.html#e-mail-state-(type=email) https://html.spec.whatwg.org/multipage/forms.html#e-mail-sta...
- wyldfire 11y ago
  >Since this is an official W3C doc, I see no reason why people shouldn't use this. Well, as far as authority/canon goes, it's typically dictated by the IETF and not W3. And, sure, they could cooperate with one another but, really -- if there's an authority on the protocols that describe email (SMTP, POP, IMAP, etc) -- it shouldn't be the World Wide Web Consortium. That said, the IETF does tend to draft RFCs that reflect actual implementations (at least their intended design), but since they often bias towards interoperability, it's unlikely they'd narrow the scope of the email address grammar.
- oinksoft 11y ago
  Practically, those W3C/WHATWG regexes probably are fine, but they aren't correct. For instance they fail to match the valid address c."@".t@gmail.com because they don't handle quoted strings correctly. I'd expect they also match some invalid addresses. My understanding was that only a Perl regex can test an address against the standard: http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html
  - Dylan16807 11y ago
    It may be incorrect to reject that email, but it's good to reject that email.
    - __david__ 11y ago
      Why reject a perfectly valid email? Why do you get to thumb your nose at that particular one?
      - Dylan16807 11y ago
        It's only 'valid' because of the particular decisions made by someone writing a random RFC. There's next to no reason to allow quoted strings, and even less reason to allow a different character set inside of them. Email addresses do not need escape sequences.
        mikeash 11y ago
        Doesn't this sort of RFC typically summarize and codify real-world usage, rather than coming up with new stuff? If a feature is in there, I'd have thought it's because somebody uses it.
        pc86 11y ago
        At some point you just have to reach the level of "Yeah... don't do that." I would suspect have "@" as part of your email probably approaches that line.
- AceJohnny2 11y ago
  The spec of what HTML will accept for an email in an input form is quite different from what email (which predates HTML by a lot, the major RFC 822 being from 1982) actually accepts.
- zurn 11y ago
  The email input regexp is only about matching the atext@domain subpart of the syntax. The full email address parsers are about parsing the full allowed syntax you can put in an email to: field that includes address lists and display names. Also W3C produced HTML specs are hardly gospel about email-related things (but I don't know whether there's anything wrong in this case).
  - kccqzy 11y ago
    But the question is, if a signup form is asking for your email address, will you type in your name, a pair of angle brackets, and then your email address in between? That's simply not what the web developer wants here.
    - lmm 11y ago
      Will I type that in? No. Will I copy-paste my email from my address book program? Quite possibly.
  - zurn 11y ago
    Replying to myself (can't edit anymore), looks like the W3C spec bungled it in your [1] by referencing "atext" from the RFC. For example, as specified it disallows addresses with periods in them. In the RFC, "atext" is just a building block in the syntax of the local-part before @ sign.
- taspeotis 11y ago
  The .NET Framework's email address validation has a pretty ... comprehensive ... regular expression [1]. [1] http://referencesource.microsoft.com/#System.ComponentModel.DataAnnotations/DataAnnotations/EmailAddressAttribute.cs,54 http://referencesource.microsoft.com/#System.ComponentModel....
  - chris_wot 11y ago
    Holy line of cartoon swear words Batman! I would hate to be the programmer who had to debug that regex.
    - leviathan 11y ago
      You clearly haven't seen the regex that matches emails according to RFC822: Behold: http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html
      - chris_wot 11y ago
        /me falls off chair
- djur 11y ago
  The W3C and WHATWG are web standards bodies. Email isn't in their bailiwick. The acceptable format for email addresses is defined in IETF RFCs.
- seszett 11y ago
  This forces users who have internationalized email addresses (say whatever@maré-design.fr - just a random IDN I know) to lookup the punycode-encoded version of the domain. Ditto for internationalized TLDs like something@taiwan.台灣). I'm not sure if this is really that bad, but I don't like it.
  - kijin 11y ago
    It's the job of the validation engine to convert all URLs and email addresses to a canonical form -- either punycode or actual UTF-8, but never both. I prefer standardizing on punycode because it's easier to handle. Otherwise you're just asking for trouble when the same user who signed up with an email address at maré-design.fr later tries to reset their password with an email address at xn--mar-design-d7a.fr. Sorry, you don't seem to have an account with us. The same thing happens with URLs. Some browsers send the Host: header in punycode but send the referer in UTF-8. Who knows how they encode CORS headers and all the other newfangled stuff that contains bits and pieces of URLs. You have to consistently convert one to the other before using any of them. IDNs are a mess.
    - seszett 11y ago
      > It's the job of the validation engine to convert all URLs and email addresses to a canonical form I don't think <input type="email"> does that though. Maybe it should, but users might be surprised to see their address automatically turn into some ugly unreadable xn--whatver-d7a domain. After form submission then yes of course any sane process will convert them to a canonical form. I know there were good reasons to use punycode instead of UTF8 for IDNs, but it sure is a mess.
      - kijin 11y ago
        Yeah, I think we'll have to treat email addresses as just another kind of untrustworthy input that needs to be escaped in various ways depending on the context. Punycode for storage and validation, back to UTF-8 for display, back to punycode when it goes in a mailto: link, etc. Ditto for URLs. The days of easy parsing are sadly forever gone.
- StavrosK 11y ago
  Saying "it can't even be matched with a regex" is trivial. Here's another, very simple thing that can't be matched with a regex: Matching pairs of nested parentheses.
hitekker 11y ago
direct link to image visualizing the complexity of the state machine required: https://i.stack.imgur.com/SrUwP.png https://i.stack.imgur.com/SrUwP.png
eli 11y ago
I think many people are thinking about email address validation the wrong way. RFC 822 describes how messages are encoded when email servers talk to each other. It isn't really about email address validation and is not intended to be used to validate a form field on some registration page. Unless you're writing an MTA or similar piece of infrastructure there is no reason you should be using the RFC grammar. Even if implementation were easy, it probably isn't what you want. For example, the spec permits inline comments but that's a nonsensical thing to have in the middle of an address you typed into an HTML form. Email addresses entered on a web form should be rejected if they contain comments, IMHO. I think what most developers really want to know is something like: Can this given email address receive messages? Or: Does this given address actually belong to this user? Well, the only way to test that is to send it a message. At best, regex validation might warn you earlier that a given address couldn't possibly work because it's so obviously malformed. But you can't validate your way into getting people to enter their real email address if they don't want to or if they don't know what it is. If your intent is really just to help catch typos and mistakes, you'd be much better off looking to something like mailcheck [0] which will flag common typos like "foo@hotnail.com" even if they result in valid looking addresses. [0] https://github.com/mailcheck/mailcheck https://github.com/mailcheck/mailcheck
- saurik 11y ago
  I agree with you "100% in spirit", but feel the need to point out that RFC 5322 (822 is ancient...) is for MIME, and thereby isn't what is even used "when e-mail servers talk to each other": it is the format of the e-mail message itself that is used only within a mailbox (when either your client or, if you are using IMAP, your server parses the message). The actual standard used by e-mail servers talking to each other is 5321 (which you might still know 821 ;P), the standard for SMTP: this protocol actually has a different way to write comments and escape characters, as it is embedded into a different structure (which I think decimates the arguments people tend to make that you should validate comments). Years ago I was working quite in earnest on an e-mail server suite, and at the time I was extremely deep in the various standards, and wrote a comment that goes into somewhat more depth on the semantics of e-mail verification. The example I was really happy with is the notion that you would never ask your user to HTML escape their username or password ;P. https://news.ycombinator.com/item?id=4486872 https://news.ycombinator.com/item?id=4486872

kccqzy 11y ago

I find that quite unbelievable. When I had a similar problem last year, the first resource I found was a W3C specification[1] about <input type=email>. The specification clearly states that email addresses should match: /^[a-zA-Z0-9.!#$%&’*+/=?^_`{|}~-]+@[a-zA-Z0-9-]+(?:\.[a-zA-Z0-9-]+)*$/ Since this is an official W3C doc, I see no reason why people shouldn't use this. Edit: There is also a version by WHATWG[2] here: /^[a-zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/ It apparently does a more thorough validation than the W3C one in the domain part, but the difference between the two is not apparent in practice. [1]: http://www.w3.org/TR/html-markup/input.email.html http://www.w3.org/TR/html-markup/input.email.html [2]: https://html.spec.whatwg.org/multipage/forms.html#e-mail-state-(type=email) https://html.spec.whatwg.org/multipage/forms.html#e-mail-sta...

wyldfire 11y ago
>Since this is an official W3C doc, I see no reason why people shouldn't use this. Well, as far as authority/canon goes, it's typically dictated by the IETF and not W3. And, sure, they could cooperate with one another but, really -- if there's an authority on the protocols that describe email (SMTP, POP, IMAP, etc) -- it shouldn't be the World Wide Web Consortium. That said, the IETF does tend to draft RFCs that reflect actual implementations (at least their intended design), but since they often bias towards interoperability, it's unlikely they'd narrow the scope of the email address grammar.
oinksoft 11y ago
Practically, those W3C/WHATWG regexes probably are fine, but they aren't correct. For instance they fail to match the valid address c."@".t@gmail.com because they don't handle quoted strings correctly. I'd expect they also match some invalid addresses. My understanding was that only a Perl regex can test an address against the standard: http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html
- Dylan16807 11y ago
  It may be incorrect to reject that email, but it's good to reject that email.
  - __david__ 11y ago
    Why reject a perfectly valid email? Why do you get to thumb your nose at that particular one?
    - Dylan16807 11y ago
      It's only 'valid' because of the particular decisions made by someone writing a random RFC. There's next to no reason to allow quoted strings, and even less reason to allow a different character set inside of them. Email addresses do not need escape sequences.
      - mikeash 11y ago
        Doesn't this sort of RFC typically summarize and codify real-world usage, rather than coming up with new stuff? If a feature is in there, I'd have thought it's because somebody uses it.
        pc86 11y ago
        At some point you just have to reach the level of "Yeah... don't do that." I would suspect have "@" as part of your email probably approaches that line.
AceJohnny2 11y ago
The spec of what HTML will accept for an email in an input form is quite different from what email (which predates HTML by a lot, the major RFC 822 being from 1982) actually accepts.
zurn 11y ago
The email input regexp is only about matching the atext@domain subpart of the syntax. The full email address parsers are about parsing the full allowed syntax you can put in an email to: field that includes address lists and display names. Also W3C produced HTML specs are hardly gospel about email-related things (but I don't know whether there's anything wrong in this case).
- kccqzy 11y ago
  But the question is, if a signup form is asking for your email address, will you type in your name, a pair of angle brackets, and then your email address in between? That's simply not what the web developer wants here.
  - lmm 11y ago
    Will I type that in? No. Will I copy-paste my email from my address book program? Quite possibly.
- zurn 11y ago
  Replying to myself (can't edit anymore), looks like the W3C spec bungled it in your [1] by referencing "atext" from the RFC. For example, as specified it disallows addresses with periods in them. In the RFC, "atext" is just a building block in the syntax of the local-part before @ sign.
taspeotis 11y ago
The .NET Framework's email address validation has a pretty ... comprehensive ... regular expression [1]. [1] http://referencesource.microsoft.com/#System.ComponentModel.DataAnnotations/DataAnnotations/EmailAddressAttribute.cs,54 http://referencesource.microsoft.com/#System.ComponentModel....
- chris_wot 11y ago
  Holy line of cartoon swear words Batman! I would hate to be the programmer who had to debug that regex.
  - leviathan 11y ago
    You clearly haven't seen the regex that matches emails according to RFC822: Behold: http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html
    - chris_wot 11y ago
      /me falls off chair
djur 11y ago
The W3C and WHATWG are web standards bodies. Email isn't in their bailiwick. The acceptable format for email addresses is defined in IETF RFCs.
seszett 11y ago
This forces users who have internationalized email addresses (say whatever@maré-design.fr - just a random IDN I know) to lookup the punycode-encoded version of the domain. Ditto for internationalized TLDs like something@taiwan.台灣). I'm not sure if this is really that bad, but I don't like it.
- kijin 11y ago
  It's the job of the validation engine to convert all URLs and email addresses to a canonical form -- either punycode or actual UTF-8, but never both. I prefer standardizing on punycode because it's easier to handle. Otherwise you're just asking for trouble when the same user who signed up with an email address at maré-design.fr later tries to reset their password with an email address at xn--mar-design-d7a.fr. Sorry, you don't seem to have an account with us. The same thing happens with URLs. Some browsers send the Host: header in punycode but send the referer in UTF-8. Who knows how they encode CORS headers and all the other newfangled stuff that contains bits and pieces of URLs. You have to consistently convert one to the other before using any of them. IDNs are a mess.
  - seszett 11y ago
    > It's the job of the validation engine to convert all URLs and email addresses to a canonical form I don't think <input type="email"> does that though. Maybe it should, but users might be surprised to see their address automatically turn into some ugly unreadable xn--whatver-d7a domain. After form submission then yes of course any sane process will convert them to a canonical form. I know there were good reasons to use punycode instead of UTF8 for IDNs, but it sure is a mess.
    - kijin 11y ago
      Yeah, I think we'll have to treat email addresses as just another kind of untrustworthy input that needs to be escaped in various ways depending on the context. Punycode for storage and validation, back to UTF-8 for display, back to punycode when it goes in a mailto: link, etc. Ditto for URLs. The days of easy parsing are sadly forever gone.
StavrosK 11y ago
Saying "it can't even be matched with a regex" is trivial. Here's another, very simple thing that can't be matched with a regex: Matching pairs of nested parentheses.

hitekker 11y ago

direct link to image visualizing the complexity of the state machine required: https://i.stack.imgur.com/SrUwP.png https://i.stack.imgur.com/SrUwP.png

eli 11y ago

I think many people are thinking about email address validation the wrong way. RFC 822 describes how messages are encoded when email servers talk to each other. It isn't really about email address validation and is not intended to be used to validate a form field on some registration page. Unless you're writing an MTA or similar piece of infrastructure there is no reason you should be using the RFC grammar. Even if implementation were easy, it probably isn't what you want. For example, the spec permits inline comments but that's a nonsensical thing to have in the middle of an address you typed into an HTML form. Email addresses entered on a web form should be rejected if they contain comments, IMHO. I think what most developers really want to know is something like: Can this given email address receive messages? Or: Does this given address actually belong to this user? Well, the only way to test that is to send it a message. At best, regex validation might warn you earlier that a given address couldn't possibly work because it's so obviously malformed. But you can't validate your way into getting people to enter their real email address if they don't want to or if they don't know what it is. If your intent is really just to help catch typos and mistakes, you'd be much better off looking to something like mailcheck [0] which will flag common typos like "foo@hotnail.com" even if they result in valid looking addresses. [0] https://github.com/mailcheck/mailcheck https://github.com/mailcheck/mailcheck

saurik 11y ago
I agree with you "100% in spirit", but feel the need to point out that RFC 5322 (822 is ancient...) is for MIME, and thereby isn't what is even used "when e-mail servers talk to each other": it is the format of the e-mail message itself that is used only within a mailbox (when either your client or, if you are using IMAP, your server parses the message). The actual standard used by e-mail servers talking to each other is 5321 (which you might still know 821 ;P), the standard for SMTP: this protocol actually has a different way to write comments and escape characters, as it is embedded into a different structure (which I think decimates the arguments people tend to make that you should validate comments). Years ago I was working quite in earnest on an e-mail server suite, and at the time I was extremely deep in the various standards, and wrote a comment that goes into somewhat more depth on the semantics of e-mail verification. The example I was really happy with is the notion that you would never ask your user to HTML escape their username or password ;P. https://news.ycombinator.com/item?id=4486872 https://news.ycombinator.com/item?id=4486872