This was a challenge for blekko when we built our search engine. A major component to ranking a URL is the cumulative rank of the domain it is hosted on. You do

This was a challenge for blekko when we built our search engine. A major component to ranking a URL is the cumulative rank of the domain it is hosted on. You don't wan't URL ranking to leak between independent sites sharing the same top level domain. Blog hosting sites are a prime example. Another factor was budgeting crawl resources. The crawler has a lot of pages to crawl and if you let it, it'd just do a deep dive on amazon.com and never come back. So dividing crawl budget between domains and subdomains is important. As noted in the article, it can be gamed so you have to guard against that algorithmically and occasionally make special cases. Prior to the Public Suffix List (https://publicsuffix.org/ https://publicsuffix.org/) project I was using the Mozilla project's list of top level domains. I had code that would download the latest TLD list from Mozilla (and later the publicsuffix.org) and generate a trie that the blekko crawler could traverse to obtain the TLD and the subdomain. Blekko kept all sorts of data on every domain and subdomain, including (but not limited to) domain rank, host IPs, country, language, average porn score per page, etc. My Perl TLD parsing code is here: https://github.com/randomstring/Net-Domain-PublicSuffix https://github.com/randomstring/Net-Domain-PublicSuffix Interesting aside: the TLD .US (https://en.wikipedia.org/wiki/.us https://en.wikipedia.org/wiki/.us) has some "weird" rules that must have seemed like a good idea when they were proposed, but complicate parsing rules. For instance in the hostname town.windermere.fl.us the "town" is significant to differentiate it from a potentially different entity hosted on co.windermere.fl.us (the county of Winermere Florida, if such a thing exists). Thankfully, few cities/counties/villages/etc use theses convoluted .us domains. Many opt for a more traditional TLD (seattle.gov for instance).