6 ms·

The IANA TLD list should never be used directly. What you really want instead is the Public Suffix List [1]. It will help you determine the "effective TLD" of d

by mike_d 6y ago

The IANA TLD list should never be used directly. What you really want instead is the Public Suffix List [1]. It will help you determine the "effective TLD" of domains like amazon.co.uk or sflawlib.ci.sf.ca.us, and gives you more insight in to how a technical allocation at the root transforms politically in to implementation. 1. https://publicsuffix.org/list/public_suffix_list.dat https://publicsuffix.org/list/public_suffix_list.dat

darnir 6y ago
And then also, please don't parse the PSL yourself. Use an existing library[1] for that. The formal algorithm for finding the TLD is more complicated than you realise at first glance. I would recommend libPSL[2]. It was written by one of the maintainers of GNU Wget and is currently used by both GNU Wget and libcurl. Disclosure: I was one of the early contributors to libpsl and closely involved in its initial formation. [1] https://publicsuffix.org/learn/ https://publicsuffix.org/learn/ [2] https://github.com/rockdaboot/libpsl https://github.com/rockdaboot/libpsl
- CodesInChaos 6y ago
  In what way is it more complicated that one would naively expect? And why?
  - exikyut 6y ago
    I'm not sure. https://github.com/rockdaboot/libpsl/blob/master/src/psl.c https://github.com/rockdaboot/libpsl/blob/master/src/psl.c is 2K lines, most of it appears to deal with memory management, character twiddling, and processing/conversion. Maybe the cookie management?
    - knome 6y ago
      For one, it appears to handle punycode internally. Punycode is a method for encoding international unicode into ascii prefixed with XN--. So this will correctly associate cookies for either the unicode term or its punycode ascii equivalent. As exanples, wikipedia indicates the international domains with the most name registrations as being the russia's "рф", taiwan's "台灣" and china's "中国", which are represented in DNS as "xn--p1ai", "xn--kpry57d", and "xn--fiqs8s", respectively. The data appears to indicate hosting sites where users can register their own names against a providers domain ( username.example.com ) as well as exceptions to this where the host's own site then uses subdomains ( www.example.com, admin.example.com, cdn.example.com ) and the host's cookies should still be used. It lists specific tlds, wildcards where appropriate with *, and notes exceptions to wildcarding by prefixing with !. Certainly far from something that would be impossible to write your own parser for, but getting everything right on your first go would be harder than one might expect, and getting things wrong here would be likely to leak the user's information between various sites.
    - sleevi 6y ago
      https://publicsuffix.org/list/ https://publicsuffix.org/list/ (See the Algorithm) Is actually hard to implement correctly and interoperably, even among browsers, and there are sharp edge cases along the way (such as holes within domain trees). The author of the library referenced at least worked with the PSL maintainers and browsers to make sure they were faithfully and correctly implementing things :)
  - ficklepickle 6y ago
    Here's one for ya. Legacy third/fourth level domains on ".ca". They were provincial/municipal respectively. Although new issuance was discontinued in 2010, they still exist. For example, posts from ansuz.sooke.bc.ca have been popular on HN. Sooke is a municipality, BC is a province. https://en.wikipedia.org/wiki/.ca#Third-level_(provincial)_and_fourth-level_(municipal)_domains https://en.wikipedia.org/wiki/.ca#Third-level_(provincial)_a...
quotemstr 6y ago
I'm glad the PSL exists, but it's a really ugly hack. The public-ness of a domain should be a queryable DNS record. We shouldn't make tons of programs bundle what's essentially an out of band zone file when we could just use DNS as intended. Yes, I know about the PSL DNS query service. That's only marginally better. The public suffix flag should be a record on the domain itself.
- schoen 6y ago
  There is a concluded IETF WG called dbound (I don't know why it concluded!) and a current Internet-Draft about doing this: https://datatracker.ietf.org/doc/draft-levine-dbound-dns/ https://datatracker.ietf.org/doc/draft-levine-dbound-dns/
  - sleevi 6y ago
    It concluded because, despite everyone agreeing they wanted a unicorn, they couldn’t agree which breed of unicorn they wanted, and thus were unable to get one. Which is to say: things went wrong in circles because different folks had different problems, those different problems had incompatible requirements, and so things spun in circles for a number of years as every idea failed to solve the problem for everyone simultaneously, while narrowly-specified solutions were shot down for not solving enough problems.
    - schoen 6y ago
      Is it still plausible that the remaining I-D will advance to an RFC?
      - sleevi 6y ago
        Perhaps? But I’m not aware of anyone (besides the DMARC crew) who could implement this, or even having been involved, so it seems mostly moot?
ricardo81 6y ago
Thankfully it exists and seems to be about one of the only sources in identifying 2nd and lower level TLDs. I remember reading how browsers were accepting cookies that'd be sent to many other sites on the same host level because the browsers had no idea on what authority level to differentiate.
- JdeBP 6y ago
  "2nd level top-level domain"? (-:
  - ricardo81 6y ago
    Hehe, in my travels I haven't found a good moniker for them, as many people would call "co.uk" a TLD. Maybe the T can stand for "toppish".
    - ZWoz 6y ago
      https://en.wikipedia.org/wiki/Top-level_domain https://en.wikipedia.org/wiki/Top-level_domain - TLD is first level[1]. Domain directly on root. Domains like co.uk and so on are second level domains. ----- [1] - or second, then again counting root as level 0 seems reasonable.
      - ricardo81 6y ago
        Yes, apparently "SLD" is quite a common term but it does not encapsulate the likes of 'pvt.k12.ma.us' or 'uk.com' (in the sense that it's not a registry per se for the latter) So there's an awkward way of defining things in various contexts, like cookie permissions, whether a domain is registerable, whether it is a registry the next level up
        ZWoz 6y ago
        uk.com is second level, pvt.k12.ma.us fourth. I think you really look words for denoting political separation (instead technical), but I don't know any good terminology for that. Public suffix is probably best term to differentiate between "shared" and "dedicated" namespace. Technically though, root servers have information only about TLD-s. co.uk may be special for some people, but a.root-servers.net. don't care, it's still sends you to uk TLD nameservers, like for every other second level domain. That political separation or what counts as "registry" is itself very tricky. Are dynamic DNS services registries? How about hosting (like blog) service provider, who gives subdomain/hostname for every client?
1vuio0pswjnm7 6y ago
"... transforms politically in to implementations." Is this list controlled by Mozilla. Or perhaps some group of browser oranisations/companies. Personally, not speaking for any other user, I am not really a fan of the browser deciding what is or is not an "acceptable" TLD, because the browser is not the only program I use for generating and sending HTTP. I use a variety of programs. Perhaps if I have some control over the browser's list. For example, I the user can add or subtract "TLDs". In the past I have done this by running an edited copy of the root.zone on the local network. I think it is a cleaner, less application-specific, solution than relying on a list compiled by a browser vendor(s). Browsers can easily override the "IANA TLD list" as well as the DNS I set up on the local network. I am not saying they are doing this through this list, but the capability is there. Browsers like Firefox are certainly not shy about constantly manipulating how domains and urls display in an address bar, Chrome wants to "protect" the user from "evil" pages, etc. It is a slippery slope. I like the idea of overriding the IANA but not the idea of this being outside the control of the user, decided by some browser vendor(s). I do not want/need applications making decisions about what is or is not a TLD, or in this case what is a legitimate subdomain for purposes of cookies. I already do that through control over the zone files I serve and system resolver settings; I control, i.e. filter, cookies through a local proxy. The root.zone has grown exponentially and is full of cruft now thanks to the "gTLD" scheme, as others have noted. If you really care about this stuff, I don't think you can rely on someone else to address the problem for you. Mozilla or whomever produces the "public suffix list" is no doubt tied to the online ad industry in some way, directly or indirectly.
- 1f60c 6y ago
  The public suffix list makes sure evil.github.io can’t read bob.github.io’s cookies, nothing more and nothing less.
  - Spivak 6y ago
    Which kinda sucks because this is functionality that should be supported but and not require a global list to work. I should be able to set origin policy on any domain I control.
  - 6y ago
sleevi 6y ago
PSL maintainer here: please don’t use the PSL! Yes, it’s weird to have a maintainer asking people not to use their project, but the PSL was a very specific (and unfortunate) hack for a very specific (and unfortunate, and browser-created) problem. It is something we live with, not something we like. While the ideal world is “don’t use any list at all, use the protocols as God, the IETF, and IANA intended”, if you are going to use a list, using the IANA list, updated daily, is much better than the PSL. Do not use the PSL for anything that is not “cookies abusing the Host header” https://github.com/sleevi/psl-problems https://github.com/sleevi/psl-problems
- methyl 6y ago
  It has pretty strong usecase in SEO where you want to determine number of unique root domains that are linking to some page.
- Lvl999Noob 6y ago
  Are you still adding suffixes to the list? If so, wouldn't refusing to add new suffixes help with the issue? If no new organisation can make use of PSL to link their subdomains, then they are only left with SOP. Since the list stays like it is now, no existing websites, depending on the list suddenly break down.
  - sleevi 6y ago
    We are. Deliberate sabotage like that would take quite a while before it was noticed, however, and it wouldn’t magically fix cookies and how people use them. To the extent it is used by cookies, we still want to maintain a fair and equitable solution. However, we also want to actively discourage any new users or use cases, to the extent possible, while we also try to fix cookies. Ideas like https://github.com/privacycg/first-party-sets https://github.com/privacycg/first-party-sets provide a possible model. While FPS doesn’t directly address this, as part of keeping a narrow scope, the approach to explicitly expressing boundaries is one that has the best viable path. However, that’s effectively “Deprecate the Host option for cookies”, so... that’s a big task. Simply sabotaging the PSL doesn’t force the problem to be solved, so mostly, it’s an education campaign of “We made a mistake; learn from ours, rather than repeating it.”
ZWoz 6y ago
Thats not really all inclusive list. Several second level domains are missing. Probably because some point it's pretty hazy, what is "public" suffix.
- neallindsay 6y ago
  > Several second level domains are missing. Second level domains are by definition not top level domains. -edit- Ha–I thought this was a top-level comment. I think that counts as irony.
  - ZWoz 6y ago
    Whole point of publicsuffix.org is list second or third level domains, which are public. Otherwise we are back to IANA list, which is indeed TLD list.

by mike_d 6y ago

darnir 6y ago

And then also, please don't parse the PSL yourself. Use an existing library[1] for that. The formal algorithm for finding the TLD is more complicated than you realise at first glance. I would recommend libPSL[2]. It was written by one of the maintainers of GNU Wget and is currently used by both GNU Wget and libcurl. Disclosure: I was one of the early contributors to libpsl and closely involved in its initial formation. [1] https://publicsuffix.org/learn/ https://publicsuffix.org/learn/ [2] https://github.com/rockdaboot/libpsl https://github.com/rockdaboot/libpsl

CodesInChaos 6y ago
In what way is it more complicated that one would naively expect? And why?
- exikyut 6y ago
  I'm not sure. https://github.com/rockdaboot/libpsl/blob/master/src/psl.c https://github.com/rockdaboot/libpsl/blob/master/src/psl.c is 2K lines, most of it appears to deal with memory management, character twiddling, and processing/conversion. Maybe the cookie management?
  - knome 6y ago
    For one, it appears to handle punycode internally. Punycode is a method for encoding international unicode into ascii prefixed with XN--. So this will correctly associate cookies for either the unicode term or its punycode ascii equivalent. As exanples, wikipedia indicates the international domains with the most name registrations as being the russia's "рф", taiwan's "台灣" and china's "中国", which are represented in DNS as "xn--p1ai", "xn--kpry57d", and "xn--fiqs8s", respectively. The data appears to indicate hosting sites where users can register their own names against a providers domain ( username.example.com ) as well as exceptions to this where the host's own site then uses subdomains ( www.example.com, admin.example.com, cdn.example.com ) and the host's cookies should still be used. It lists specific tlds, wildcards where appropriate with *, and notes exceptions to wildcarding by prefixing with !. Certainly far from something that would be impossible to write your own parser for, but getting everything right on your first go would be harder than one might expect, and getting things wrong here would be likely to leak the user's information between various sites.
  - sleevi 6y ago
    https://publicsuffix.org/list/ https://publicsuffix.org/list/ (See the Algorithm) Is actually hard to implement correctly and interoperably, even among browsers, and there are sharp edge cases along the way (such as holes within domain trees). The author of the library referenced at least worked with the PSL maintainers and browsers to make sure they were faithfully and correctly implementing things :)
- ficklepickle 6y ago
  Here's one for ya. Legacy third/fourth level domains on ".ca". They were provincial/municipal respectively. Although new issuance was discontinued in 2010, they still exist. For example, posts from ansuz.sooke.bc.ca have been popular on HN. Sooke is a municipality, BC is a province. https://en.wikipedia.org/wiki/.ca#Third-level_(provincial)_and_fourth-level_(municipal)_domains https://en.wikipedia.org/wiki/.ca#Third-level_(provincial)_a...

quotemstr 6y ago

I'm glad the PSL exists, but it's a really ugly hack. The public-ness of a domain should be a queryable DNS record. We shouldn't make tons of programs bundle what's essentially an out of band zone file when we could just use DNS as intended. Yes, I know about the PSL DNS query service. That's only marginally better. The public suffix flag should be a record on the domain itself.

schoen 6y ago
There is a concluded IETF WG called dbound (I don't know why it concluded!) and a current Internet-Draft about doing this: https://datatracker.ietf.org/doc/draft-levine-dbound-dns/ https://datatracker.ietf.org/doc/draft-levine-dbound-dns/
- sleevi 6y ago
  It concluded because, despite everyone agreeing they wanted a unicorn, they couldn’t agree which breed of unicorn they wanted, and thus were unable to get one. Which is to say: things went wrong in circles because different folks had different problems, those different problems had incompatible requirements, and so things spun in circles for a number of years as every idea failed to solve the problem for everyone simultaneously, while narrowly-specified solutions were shot down for not solving enough problems.
  - schoen 6y ago
    Is it still plausible that the remaining I-D will advance to an RFC?
    - sleevi 6y ago
      Perhaps? But I’m not aware of anyone (besides the DMARC crew) who could implement this, or even having been involved, so it seems mostly moot?

ricardo81 6y ago

Thankfully it exists and seems to be about one of the only sources in identifying 2nd and lower level TLDs. I remember reading how browsers were accepting cookies that'd be sent to many other sites on the same host level because the browsers had no idea on what authority level to differentiate.

JdeBP 6y ago
"2nd level top-level domain"? (-:
- ricardo81 6y ago
  Hehe, in my travels I haven't found a good moniker for them, as many people would call "co.uk" a TLD. Maybe the T can stand for "toppish".
  - ZWoz 6y ago
    https://en.wikipedia.org/wiki/Top-level_domain https://en.wikipedia.org/wiki/Top-level_domain - TLD is first level[1]. Domain directly on root. Domains like co.uk and so on are second level domains. ----- [1] - or second, then again counting root as level 0 seems reasonable.
    - ricardo81 6y ago
      Yes, apparently "SLD" is quite a common term but it does not encapsulate the likes of 'pvt.k12.ma.us' or 'uk.com' (in the sense that it's not a registry per se for the latter) So there's an awkward way of defining things in various contexts, like cookie permissions, whether a domain is registerable, whether it is a registry the next level up
      - ZWoz 6y ago
        uk.com is second level, pvt.k12.ma.us fourth. I think you really look words for denoting political separation (instead technical), but I don't know any good terminology for that. Public suffix is probably best term to differentiate between "shared" and "dedicated" namespace. Technically though, root servers have information only about TLD-s. co.uk may be special for some people, but a.root-servers.net. don't care, it's still sends you to uk TLD nameservers, like for every other second level domain. That political separation or what counts as "registry" is itself very tricky. Are dynamic DNS services registries? How about hosting (like blog) service provider, who gives subdomain/hostname for every client?

1vuio0pswjnm7 6y ago

"... transforms politically in to implementations." Is this list controlled by Mozilla. Or perhaps some group of browser oranisations/companies. Personally, not speaking for any other user, I am not really a fan of the browser deciding what is or is not an "acceptable" TLD, because the browser is not the only program I use for generating and sending HTTP. I use a variety of programs. Perhaps if I have some control over the browser's list. For example, I the user can add or subtract "TLDs". In the past I have done this by running an edited copy of the root.zone on the local network. I think it is a cleaner, less application-specific, solution than relying on a list compiled by a browser vendor(s). Browsers can easily override the "IANA TLD list" as well as the DNS I set up on the local network. I am not saying they are doing this through this list, but the capability is there. Browsers like Firefox are certainly not shy about constantly manipulating how domains and urls display in an address bar, Chrome wants to "protect" the user from "evil" pages, etc. It is a slippery slope. I like the idea of overriding the IANA but not the idea of this being outside the control of the user, decided by some browser vendor(s). I do not want/need applications making decisions about what is or is not a TLD, or in this case what is a legitimate subdomain for purposes of cookies. I already do that through control over the zone files I serve and system resolver settings; I control, i.e. filter, cookies through a local proxy. The root.zone has grown exponentially and is full of cruft now thanks to the "gTLD" scheme, as others have noted. If you really care about this stuff, I don't think you can rely on someone else to address the problem for you. Mozilla or whomever produces the "public suffix list" is no doubt tied to the online ad industry in some way, directly or indirectly.

1f60c 6y ago
The public suffix list makes sure evil.github.io can’t read bob.github.io’s cookies, nothing more and nothing less.
- Spivak 6y ago
  Which kinda sucks because this is functionality that should be supported but and not require a global list to work. I should be able to set origin policy on any domain I control.
- 6y ago

sleevi 6y ago

PSL maintainer here: please don’t use the PSL! Yes, it’s weird to have a maintainer asking people not to use their project, but the PSL was a very specific (and unfortunate) hack for a very specific (and unfortunate, and browser-created) problem. It is something we live with, not something we like. While the ideal world is “don’t use any list at all, use the protocols as God, the IETF, and IANA intended”, if you are going to use a list, using the IANA list, updated daily, is much better than the PSL. Do not use the PSL for anything that is not “cookies abusing the Host header” https://github.com/sleevi/psl-problems https://github.com/sleevi/psl-problems

methyl 6y ago
It has pretty strong usecase in SEO where you want to determine number of unique root domains that are linking to some page.
Lvl999Noob 6y ago
Are you still adding suffixes to the list? If so, wouldn't refusing to add new suffixes help with the issue? If no new organisation can make use of PSL to link their subdomains, then they are only left with SOP. Since the list stays like it is now, no existing websites, depending on the list suddenly break down.
- sleevi 6y ago
  We are. Deliberate sabotage like that would take quite a while before it was noticed, however, and it wouldn’t magically fix cookies and how people use them. To the extent it is used by cookies, we still want to maintain a fair and equitable solution. However, we also want to actively discourage any new users or use cases, to the extent possible, while we also try to fix cookies. Ideas like https://github.com/privacycg/first-party-sets https://github.com/privacycg/first-party-sets provide a possible model. While FPS doesn’t directly address this, as part of keeping a narrow scope, the approach to explicitly expressing boundaries is one that has the best viable path. However, that’s effectively “Deprecate the Host option for cookies”, so... that’s a big task. Simply sabotaging the PSL doesn’t force the problem to be solved, so mostly, it’s an education campaign of “We made a mistake; learn from ours, rather than repeating it.”

ZWoz 6y ago

Thats not really all inclusive list. Several second level domains are missing. Probably because some point it's pretty hazy, what is "public" suffix.

neallindsay 6y ago
> Several second level domains are missing. Second level domains are by definition not top level domains. -edit- Ha–I thought this was a top-level comment. I think that counts as irony.
- ZWoz 6y ago
  Whole point of publicsuffix.org is list second or third level domains, which are public. Otherwise we are back to IANA list, which is indeed TLD list.