6 ms·

The Public Suffix List

79 points by dan1234 10y ago

throwanem 10y ago
A domain hacker's playground.
- vog 10y ago
  Indeed, although I suspect that domain grabbers have their own lists anyway. This list may help to level the playfield between domain grabbers and legitimate domain users.
  - corobo 10y ago
    It's not the be all end all though. A lot of the domains listed are not usable by most people. If you've got a script searching for available domains you don't really want it wasting time searching dictionary words on *.accident-investigation.aero as they'll probably all flag legit, meaning if you use the DNS lookup then whois method you've wasted a lot of time searching for domains you can't have.
  - robin_reala 10y ago
    This list has been maintained for 9 years, if there was any leveling that needed doing it happened a long time ago.
- dan1234 10y ago
  I was thinking of using this as the basis for some email validation as I got a lot of incorrect email addresses in form posts.
  - Freak_NL 10y ago
    Please don't. A lot has been written about email address validation (search HN for a few good discussions); it is horrendously complex to get right. Any mistake or oversight means perfectly valid email addresses won't work, and maintaining your home-grown solution means keeping tabs on any updates to this domain suffix list and other relevant standards. The most sensible approach (in my opinion) is to validate with this minimal regex: /.+@.+/ Essentially just requiring an at-mark between two other bits of text. After that send a confirmation email to see if it works. That is pretty much fool-proof and low maintenance. If you absolutely must help the user in preventing any common mistakes, use a library that provides hints in the UI for those presumed errors without actually invalidating the input. There are JavaScript libraries that do this fairly well.
    - brb3 10y ago
      > Essentially just requiring an at-mark between two other bits of text. I've been using this regex to make sure there actually is text before and after the @ /.+@.+/
      - Freak_NL 10y ago
        You are right of course. A typo on my end (fixed).
      - infogulch 10y ago
        Can there be more than one @ or whitespace? If not you could use: /^[^@\s]+@[^@\s]+$/ Filtering out emails with multiple @'s and whitespace seems like an obvious win to help prevent typos and copy-paste errors.
        jwilk 10y ago
        Yes, the are valid e-mail addresses that contain whitespace or multiple @s. Wikipedia <https://en.wikipedia.org/wiki/Email_address#Valid_email_addresses https://en.wikipedia.org/wiki/Email_address#Valid_email_addr... > gives the following example: "very.(),:;<>[]\".VERY.\"very@\\ \"very\".unusual"@strange.example.com
        infogulch 10y ago
        Oh my. Nevermind. `/.+@.+/` it is.
        Freak_NL 10y ago
        Exactly. It is very much our nature as programmers to attempt to capture this seemingly trivial bit of structured text into seemingly innocuous programming rules, but the rewards of the added complexity are just too meager. Even if an email address validates, it still doesn't mean that you have a working email address. Sending an email is the only way to be sure, and if you are already doing that, then why risk excluding valid email addresses with a decidedly non-trivial regex? There are regexes out there that capture all the complexity of valid email addresses today (but who knows if they'll work with, say, next year's additions to the top-level domains?), and you can copy them from StackOverflow if you really want them; but why bother?
  - throwanem 10y ago
    Freak_NL says: > Please don't. He is correct. Seriously. Don't. The right way to validate an email address, if that's something you need to do, is by sending it a message with a link and a response code, and seeing whether the user clicks the link or enters the code. This is the only right way. There are many wrong ways. They make people unhappy. They will make you unhappy. Do not use them. You cannot validate email addresses for shape and form. The sole invariant is that there be at least three characters one of which is @. Everything beyond that is in the hands of the DNS and the domain's MTA. You cannot predict every fashion in which they will behave. You should not try to. Have you seen the regex that correctly matches every variant of email address form described in RFC 822? It is five kilobytes long. "Ah," you may now think, "I can use that!" You should not. There are new standards with new variations. The regex is incomplete. It will be incomplete forever. It is a five-kilobyte Perl regex. No one will ever understand it well enough to extend it. Just send a link and a code. Ask the user to click on the link or give you the code. When you have received the click or the code, you know the email is valid. When you have not, assume it is not. This is the method that works. It is the only method that works. Use this method and be happy. Use any other and be sad. Which you prefer is up to you.
    - snowwolf 10y ago
      I prefer to do email suggestion rather than validation - The only validation I have is that it should contain an @. And then use something like http://getmailcheck.org/ http://getmailcheck.org/ to suggest spelling corrections for common domains. We saw a massive reduction in bad emails just by implementing this without having to annoy users with incorrectly implemented validation rules.
    - geofft 10y ago
      > You cannot validate email addresses for shape and form. The sole invariant is that there be at least three characters one of which is @. Everything beyond that is in the hands of the DNS and the domain's MTA. You cannot predict every fashion in which they will behave. You should not try to. I used to agree with this viewpoint, but now I'm having some doubts. Suppose I enter the email address "geofft@example.net@example.com". It validates according to your rule. Where does it go? Suppose, furthermore, that I upgrade my servers and they start parsing it differently. They're allowed to do that, right? Maybe my old MTA sent a confirmation email to example.net, and the new one is now sending emails to a user named "geofft@example.net" at example.com. Haven't I just done something very wrong by allowing a user to trick me into sending emails to an unvalidated address? Isn't this both a violation of the Postel principle ("conservative in what you send to others") and a security hole? It seems like it would be better for my application to parse the email address and validate what it's doing, and store only unambiguously-parseable addresses. Or maybe they're not allowed to parse it differently. Maybe there's a single consistent way to parse geofft@example.net@example.com, and every single email application I might use will get it right. What is this esoteric lore that email applications know that my application cannot? If they can parse it, can't I?
  - spc476 10y ago
    If you feel you really have to validate an email address without sending an email to said address, don't use a regex. Use something that's meant for parsing, like LPeg (for example: https://github.com/spc476/LPeg-Parsers/blob/ee63df57826fd5ef62eac5669b444e5d7c9da6ff/email.lua#L504 https://github.com/spc476/LPeg-Parsers/blob/ee63df57826fd5ef... which handles the RFC grammar, including comments) or lex/yacc or something other than a regex.
cdubzzz 10y ago
I recently found this list and use it in a Drupal module to invoke hooks based on the domain of a URL. I didn't realize how many edge cases there are for these and originally tried to do it with regex.
deleted 10y ago
[deleted]
treve 10y ago
A great and important list. It also feels a bit like a hack. Would there a be a better way to do this, maybe in the DNS system to denote ownership/isolation?
- avian 10y ago
  Yes, it is a hack. I maintain the publicsuffix Python library (https://pypi.python.org/pypi/publicsuffix/ https://pypi.python.org/pypi/publicsuffix/). I had many long discussions with package and application maintainers on the topic of how to best provide data to the library. Arguably you need an up-to-date version of the list if you want to access the web securely today. You can bundle a static version with the library. That's nice for developers, and users because it just works after "pip install", but will get out of date because library won't update as often as the list changes. Some distros like Debian provide their own mechanism for updating the list. You can also download behind the scenes and cache. This way data is always up to date, but users don't like that apps call some web service on start-up and some people want to use the library off-line. If you look at different libraries and apps, each one does it differently. Getting it right is hard. Things would be so much easier if this information would come decentralized from the DNS system.
- pfg 10y ago
  There's a IETF WG working on this[1]. It's been a while since I checked on the progress, so this might not be up-to-date, but it was DNS-based, with a way to export to a static file like the PSL (I'd assume in order to avoid the performance impact of browsers having to wait for another DNS query to determine cookie scope, etc.) [1]: https://datatracker.ietf.org/wg/dbound/charter/ https://datatracker.ietf.org/wg/dbound/charter/
- spc476 10y ago
  Fix how cookies work. From my understanding, that's why this list is needed.
yrro 10y ago
I have always wondered why this information is not stored in the DNS itself.
- hannob 10y ago
  This is security critical information. DNS is not secure. (Don't say DNSSEC - practically undeployed solutions don't count as solutions.)
randomstring 10y ago
This list is used to determine what domains and sub-domains "belong" together, in the sense that they are controlled and/or owned by the same entity. For instance *.google.com is all google, so x.google.com and y.google.com can be trusted to share the same SSL key, safely share javascript (XSS), etc. However x.blogger.com and y.blogger.com are probably two completely separate blogs, people, domains, SSL keys, javascript domains, etc. And you wouldn't want to see x.blogger.com's web pages showing up in search results for y.blogger.com. Secondly, maintaining this list is a pain. You have hundreds of two letter top level domains, one for each country. Each country with its own NIC in charge of sub-domains. Each NIC with the power to add or delete subdomains (check out https://en.wikipedia.org/wiki/.uk https://en.wikipedia.org/wiki/.uk for just one of hundreds of examples). Some "countries" even sell off their top level domain like .nu . Then you have .us (https://en.wikipedia.org/wiki/.us https://en.wikipedia.org/wiki/.us) that has wild card domains like http://vil.stockbridge.mi.us/ http://vil.stockbridge.mi.us/ where the vil. is fixed and part of the domain and stockbridge.mi is the important domain information. Of course they're always adding more top level domains: .ninja, .wtf, etc (maybe there is a .etc now?). Then you have all the blogging and hosting platforms that use personalized domains for hosting content. Many are listed in the publicsuffix list, but I'm guessing not all! I ended up writing my own publicsuffix parser in PERL a few years back for the blekko search engine. The main purpose being to be able to group web pages together by site/owner. There is nothing quite like feeding every URL on the internet through your parser to find bugs and corner cases.
profmonocle 10y ago
If you use this for something, please be sure to keep the data fresh! Especially now that new TLDs are being added at a steady clip. I once had an issue where a mobile user's browser would do a Google search every time they typed one of our domains into the URL bar. They weren't doing anything wrong - other domains worked fine. They were using some Android phone from 2013 which had been abandoned by the manufacturer, and it turned out the stock browser was using the public suffix list to decide if the input should be treated as a URL or as a search query. The list was as old as the browser, and since the domain ended in ".link" (added in 2014) it didn't think our domain was a domain. (The workaround was to type out http:// http:// or https:// https:// before the domain, but that's crappy UX.) If you use this in a server-side app, please use cron or something to keep it updated. If you embed it in a client app, make sure to update it as part of your build process. And if you know your app won't be updated very often, consider having it update separately from the app itself. (Does anyone know if this is hosted on some public CDN somewhere? I assume having a client app fetch directly from publicsuffix.org isn't kosher.) Edit: My mistake, on https://publicsuffix.org/list/ https://publicsuffix.org/list/ they do recommend having client apps pull directly from their site, and limiting updates to once per day.
- waldir 10y ago
  > Especially now that new TLDs are being added at a steady clip. Indeed, there are some differences (https://www.diffchecker.com/tcbbvy7p https://www.diffchecker.com/tcbbvy7p) with IANA's root domains list: http://www.iana.org/domains/root/db http://www.iana.org/domains/root/db (which only includes TLDs, as the name indicates)
derefr 10y ago
@HN mods: this list could be used to make the domain in parens after the link more accurate, no?
jasonjei 10y ago
I do hope they use the same care and due process as they do with root certificate inclusion in the public suffix list (as well as the ability to add/remove entries manually). I think this is a great move for better security, but with great power comes great responsibility.
DanielDent 10y ago
Quick plug for my related project: https://github.com/QA2/public-suffix-metalist https://github.com/QA2/public-suffix-metalist - pull requests welcome.

79 points by dan1234 10y ago

throwanem 10y ago

A domain hacker's playground.

vog 10y ago
Indeed, although I suspect that domain grabbers have their own lists anyway. This list may help to level the playfield between domain grabbers and legitimate domain users.
- corobo 10y ago
  It's not the be all end all though. A lot of the domains listed are not usable by most people. If you've got a script searching for available domains you don't really want it wasting time searching dictionary words on *.accident-investigation.aero as they'll probably all flag legit, meaning if you use the DNS lookup then whois method you've wasted a lot of time searching for domains you can't have.
- robin_reala 10y ago
  This list has been maintained for 9 years, if there was any leveling that needed doing it happened a long time ago.
dan1234 10y ago
I was thinking of using this as the basis for some email validation as I got a lot of incorrect email addresses in form posts.
- Freak_NL 10y ago
  Please don't. A lot has been written about email address validation (search HN for a few good discussions); it is horrendously complex to get right. Any mistake or oversight means perfectly valid email addresses won't work, and maintaining your home-grown solution means keeping tabs on any updates to this domain suffix list and other relevant standards. The most sensible approach (in my opinion) is to validate with this minimal regex: /.+@.+/ Essentially just requiring an at-mark between two other bits of text. After that send a confirmation email to see if it works. That is pretty much fool-proof and low maintenance. If you absolutely must help the user in preventing any common mistakes, use a library that provides hints in the UI for those presumed errors without actually invalidating the input. There are JavaScript libraries that do this fairly well.
  - brb3 10y ago
    > Essentially just requiring an at-mark between two other bits of text. I've been using this regex to make sure there actually is text before and after the @ /.+@.+/
    - Freak_NL 10y ago
      You are right of course. A typo on my end (fixed).
    - infogulch 10y ago
      Can there be more than one @ or whitespace? If not you could use: /^[^@\s]+@[^@\s]+$/ Filtering out emails with multiple @'s and whitespace seems like an obvious win to help prevent typos and copy-paste errors.
      - jwilk 10y ago
        Yes, the are valid e-mail addresses that contain whitespace or multiple @s. Wikipedia <https://en.wikipedia.org/wiki/Email_address#Valid_email_addresses https://en.wikipedia.org/wiki/Email_address#Valid_email_addr... > gives the following example: "very.(),:;<>[]\".VERY.\"very@\\ \"very\".unusual"@strange.example.com
        infogulch 10y ago
        Oh my. Nevermind. `/.+@.+/` it is.
        Freak_NL 10y ago
        Exactly. It is very much our nature as programmers to attempt to capture this seemingly trivial bit of structured text into seemingly innocuous programming rules, but the rewards of the added complexity are just too meager. Even if an email address validates, it still doesn't mean that you have a working email address. Sending an email is the only way to be sure, and if you are already doing that, then why risk excluding valid email addresses with a decidedly non-trivial regex? There are regexes out there that capture all the complexity of valid email addresses today (but who knows if they'll work with, say, next year's additions to the top-level domains?), and you can copy them from StackOverflow if you really want them; but why bother?
- throwanem 10y ago
  Freak_NL says: > Please don't. He is correct. Seriously. Don't. The right way to validate an email address, if that's something you need to do, is by sending it a message with a link and a response code, and seeing whether the user clicks the link or enters the code. This is the only right way. There are many wrong ways. They make people unhappy. They will make you unhappy. Do not use them. You cannot validate email addresses for shape and form. The sole invariant is that there be at least three characters one of which is @. Everything beyond that is in the hands of the DNS and the domain's MTA. You cannot predict every fashion in which they will behave. You should not try to. Have you seen the regex that correctly matches every variant of email address form described in RFC 822? It is five kilobytes long. "Ah," you may now think, "I can use that!" You should not. There are new standards with new variations. The regex is incomplete. It will be incomplete forever. It is a five-kilobyte Perl regex. No one will ever understand it well enough to extend it. Just send a link and a code. Ask the user to click on the link or give you the code. When you have received the click or the code, you know the email is valid. When you have not, assume it is not. This is the method that works. It is the only method that works. Use this method and be happy. Use any other and be sad. Which you prefer is up to you.
  - snowwolf 10y ago
    I prefer to do email suggestion rather than validation - The only validation I have is that it should contain an @. And then use something like http://getmailcheck.org/ http://getmailcheck.org/ to suggest spelling corrections for common domains. We saw a massive reduction in bad emails just by implementing this without having to annoy users with incorrectly implemented validation rules.
  - geofft 10y ago
    > You cannot validate email addresses for shape and form. The sole invariant is that there be at least three characters one of which is @. Everything beyond that is in the hands of the DNS and the domain's MTA. You cannot predict every fashion in which they will behave. You should not try to. I used to agree with this viewpoint, but now I'm having some doubts. Suppose I enter the email address "geofft@example.net@example.com". It validates according to your rule. Where does it go? Suppose, furthermore, that I upgrade my servers and they start parsing it differently. They're allowed to do that, right? Maybe my old MTA sent a confirmation email to example.net, and the new one is now sending emails to a user named "geofft@example.net" at example.com. Haven't I just done something very wrong by allowing a user to trick me into sending emails to an unvalidated address? Isn't this both a violation of the Postel principle ("conservative in what you send to others") and a security hole? It seems like it would be better for my application to parse the email address and validate what it's doing, and store only unambiguously-parseable addresses. Or maybe they're not allowed to parse it differently. Maybe there's a single consistent way to parse geofft@example.net@example.com, and every single email application I might use will get it right. What is this esoteric lore that email applications know that my application cannot? If they can parse it, can't I?
- spc476 10y ago
  If you feel you really have to validate an email address without sending an email to said address, don't use a regex. Use something that's meant for parsing, like LPeg (for example: https://github.com/spc476/LPeg-Parsers/blob/ee63df57826fd5ef62eac5669b444e5d7c9da6ff/email.lua#L504 https://github.com/spc476/LPeg-Parsers/blob/ee63df57826fd5ef... which handles the RFC grammar, including comments) or lex/yacc or something other than a regex.

cdubzzz 10y ago

I recently found this list and use it in a Drupal module to invoke hooks based on the domain of a URL. I didn't realize how many edge cases there are for these and originally tried to do it with regex.

deleted 10y ago

[deleted]

treve 10y ago

A great and important list. It also feels a bit like a hack. Would there a be a better way to do this, maybe in the DNS system to denote ownership/isolation?

avian 10y ago
Yes, it is a hack. I maintain the publicsuffix Python library (https://pypi.python.org/pypi/publicsuffix/ https://pypi.python.org/pypi/publicsuffix/). I had many long discussions with package and application maintainers on the topic of how to best provide data to the library. Arguably you need an up-to-date version of the list if you want to access the web securely today. You can bundle a static version with the library. That's nice for developers, and users because it just works after "pip install", but will get out of date because library won't update as often as the list changes. Some distros like Debian provide their own mechanism for updating the list. You can also download behind the scenes and cache. This way data is always up to date, but users don't like that apps call some web service on start-up and some people want to use the library off-line. If you look at different libraries and apps, each one does it differently. Getting it right is hard. Things would be so much easier if this information would come decentralized from the DNS system.
pfg 10y ago
There's a IETF WG working on this[1]. It's been a while since I checked on the progress, so this might not be up-to-date, but it was DNS-based, with a way to export to a static file like the PSL (I'd assume in order to avoid the performance impact of browsers having to wait for another DNS query to determine cookie scope, etc.) [1]: https://datatracker.ietf.org/wg/dbound/charter/ https://datatracker.ietf.org/wg/dbound/charter/
spc476 10y ago
Fix how cookies work. From my understanding, that's why this list is needed.

yrro 10y ago

I have always wondered why this information is not stored in the DNS itself.

hannob 10y ago
This is security critical information. DNS is not secure. (Don't say DNSSEC - practically undeployed solutions don't count as solutions.)

randomstring 10y ago

This list is used to determine what domains and sub-domains "belong" together, in the sense that they are controlled and/or owned by the same entity. For instance *.google.com is all google, so x.google.com and y.google.com can be trusted to share the same SSL key, safely share javascript (XSS), etc. However x.blogger.com and y.blogger.com are probably two completely separate blogs, people, domains, SSL keys, javascript domains, etc. And you wouldn't want to see x.blogger.com's web pages showing up in search results for y.blogger.com. Secondly, maintaining this list is a pain. You have hundreds of two letter top level domains, one for each country. Each country with its own NIC in charge of sub-domains. Each NIC with the power to add or delete subdomains (check out https://en.wikipedia.org/wiki/.uk https://en.wikipedia.org/wiki/.uk for just one of hundreds of examples). Some "countries" even sell off their top level domain like .nu . Then you have .us (https://en.wikipedia.org/wiki/.us https://en.wikipedia.org/wiki/.us) that has wild card domains like http://vil.stockbridge.mi.us/ http://vil.stockbridge.mi.us/ where the vil. is fixed and part of the domain and stockbridge.mi is the important domain information. Of course they're always adding more top level domains: .ninja, .wtf, etc (maybe there is a .etc now?). Then you have all the blogging and hosting platforms that use personalized domains for hosting content. Many are listed in the publicsuffix list, but I'm guessing not all! I ended up writing my own publicsuffix parser in PERL a few years back for the blekko search engine. The main purpose being to be able to group web pages together by site/owner. There is nothing quite like feeding every URL on the internet through your parser to find bugs and corner cases.

profmonocle 10y ago

If you use this for something, please be sure to keep the data fresh! Especially now that new TLDs are being added at a steady clip. I once had an issue where a mobile user's browser would do a Google search every time they typed one of our domains into the URL bar. They weren't doing anything wrong - other domains worked fine. They were using some Android phone from 2013 which had been abandoned by the manufacturer, and it turned out the stock browser was using the public suffix list to decide if the input should be treated as a URL or as a search query. The list was as old as the browser, and since the domain ended in ".link" (added in 2014) it didn't think our domain was a domain. (The workaround was to type out http:// http:// or https:// https:// before the domain, but that's crappy UX.) If you use this in a server-side app, please use cron or something to keep it updated. If you embed it in a client app, make sure to update it as part of your build process. And if you know your app won't be updated very often, consider having it update separately from the app itself. (Does anyone know if this is hosted on some public CDN somewhere? I assume having a client app fetch directly from publicsuffix.org isn't kosher.) Edit: My mistake, on https://publicsuffix.org/list/ https://publicsuffix.org/list/ they do recommend having client apps pull directly from their site, and limiting updates to once per day.

waldir 10y ago
> Especially now that new TLDs are being added at a steady clip. Indeed, there are some differences (https://www.diffchecker.com/tcbbvy7p https://www.diffchecker.com/tcbbvy7p) with IANA's root domains list: http://www.iana.org/domains/root/db http://www.iana.org/domains/root/db (which only includes TLDs, as the name indicates)

derefr 10y ago

@HN mods: this list could be used to make the domain in parens after the link more accurate, no?

jasonjei 10y ago

I do hope they use the same care and due process as they do with root certificate inclusion in the public suffix list (as well as the ability to add/remove entries manually). I think this is a great move for better security, but with great power comes great responsibility.

DanielDent 10y ago

Quick plug for my related project: https://github.com/QA2/public-suffix-metalist https://github.com/QA2/public-suffix-metalist - pull requests welcome.