6 ms·

Show HN: I Scraped Hacker News for TLD Popularity

68 points by klausbreyer 4y ago

ryan29 4y ago
This is great! It's something I've wondered about for a while. I was surprised to see the decline of .com is fairly linear. Before I looked I was expecting the use of alternate TLDs to be accelerating a bit. Are the absolute values the running totals? If so, why do they decline from 2021 to 2022? I think a graph for unique counts would be cool to see too. For example, the ClickHouse query posted earlier in this thread shows: domain count unique ------------------------- .org 349414 58226 .net 114499 31129 So the submissions using .org are 3x .net, but the unique domains seen using .org are less than 2x .net. I'm not sure if there's any significance there, but it would be interesting to see the difference. In the same context, I think it would be interesting to see the top 50 domains on each TLD. Anyway, it's very cool info to see. Thanks for sharing it!
- klausbreyer 4y ago
  Good feedback, thank you very much! I will see if I adress this in the future. :)
testernews 4y ago
domain.com./whatever is valid tho?
- klausbreyer 4y ago
  In theory, technically it is correct. But it was harder for me to parse and it really was not a significant amount.
gryn 4y ago
you don't need to scrape HN there's the public dataset in google BigQuery, I don't know if it's still updated regularly. edit: here's the link https://console.cloud.google.com/marketplace/details/y-combinator/hacker-news https://console.cloud.google.com/marketplace/details/y-combi... they seems to have stopped updating around the later part of 2022, don't know why.
- pfarrell 4y ago
  To anyone interested, I noticed this stoppage as well. IIRC, the data in BigQuery stopped being updated Nov 16, 2022. I’ve got a complete dataset that merges in new content from the firebase dataset, then rectifies story points at a later time (since stories can amass points indefinitely). Comment here if you’re interested. I’m thinking of publishing it as a torrent, but don’t know if there’s enough interest.
  - malborodog 4y ago
    +1
  - codetrotter 4y ago
    > Comment here if you’re interested. I’m thinking of publishing it as a torrent, but don’t know if there’s enough interest. Please do :)
    - gbertb 4y ago
      Would be interested as well!
  - zxexz 4y ago
    I would love that! I find HN useful to have backed up on my laptop, for freetext searching offline mostly.
- klausbreyer 4y ago
  I also learned this today. Thanks for sharing! :) Edit: But sadly it is not up to date. :)
jossclimb 4y ago
I can't figure out what they did? Web scrapped hacker news for domain ideas?
skizm 4y ago
Dumb question: but I thought the ".io" TLD was owned by a really sketchy group / organization or ownership was being contested (or similar?) and having a domain there was semi-risky, is that still the case? I've avoided .io TLD because of this vague notion, but never really knew the specifics.
- cybrexalpha 4y ago
  If you're using .io because you want to use a "techy" TLD (that isn't .net that isn't held in high esteem for some reason), .dev is a great alternative.
  - elcritch 4y ago
    Isn't that Google's TLD? Unfortunately I trust them less than the sketchy administrative situation of .io. Wake up one day and their algo's decided you're a spam account and lock you out and take your domain.
    - flangola7 4y ago
      .tech .technology .software .it .digital are options. I agree about avoiding Google.
      - klausbreyer 4y ago
        There is also .rest (for APIs ), .run and .codes ;)
    - cybrexalpha 4y ago
      You don't need to get it directly via Google. Mine are registered through a normal registrar, and not linked to a Google Account.
- petercooper 4y ago
  The biggest concern I've seen over the .io TLD is that it's the ccTLD for a territory ("British Indian Ocean Territory") whose existence / sovereignty is fragile. The people who originally lived in this territory were forcibly expelled in the 1970s – https://en.wikipedia.org/wiki/Expulsion_of_the_Chagossians https://en.wikipedia.org/wiki/Expulsion_of_the_Chagossians – so there are problems from two points of view: ethically this doesn't sit right with a lot of folks, but even from a pragmatic point of view, the UNGA adopted a resolution in 2019 specifying that the UK should withdraw its administration from the territory – this has not yet occurred, and it is uncertain what could/would happen with the ccTLD if or when it does.
ktpsns 4y ago
The interpretation you never asked for: The data exhibit a strong preferential attachment [1] behaviour, i.e. you can draw a line in a log-log plot (despite only a semilog plot is shown). This is typical for real world data. [1] https://en.wikipedia.org/wiki/Preferential_attachment https://en.wikipedia.org/wiki/Preferential_attachment
- klausbreyer 4y ago
  Thanks! A good thing to know!
zX41ZdbW 4y ago
This will save your time: https://play.clickhouse.com/play?user=play#U0VMRUNUIHRvcExldmVsRG9tYWluKHVybCkgQVMgZG9tYWluLCBjb3VudCgpIEFTIGMsIHN1bShzY29yZSksIHVuaXEoYnkpLCByb3VuZChhdmcoc2NvcmUpLCAzKSBGUk9NIGhhY2tlcm5ld3MgV0hFUkUgdHlwZSA9ICdzdG9yeScgQU5EIGRvbWFpbiAhPSAnJyBHUk9VUCBCWSAxIE9SREVSIEJZIDIgREVTQw== https://play.clickhouse.com/play?user=play#U0VMRUNUIHRvcExld... Takeaways: - .org is 50% better than .com; - .edu and .gov are really nice; - .io is cool, and much better than .uk; PS. I don't remember any rate limits of the API. Here is how I downloaded the data: https://github.com/ClickHouse/ClickHouse/issues/29693 https://github.com/ClickHouse/ClickHouse/issues/29693
- tylerhannan 4y ago
  Thanks zX41zdbW - I find the query you shared easier to reason about than: https://github.com/klausbreyer/hackernewsstats/blob/main/queries/count-years.sql https://github.com/klausbreyer/hackernewsstats/blob/main/que... Also: "Elapsed: 0.140 sec, read 33.95 million rows, 1.26 GB." 140 ms is quite impressive.
  - return_to_monke 4y ago
    to be fair, that is large part because CH seems to have a topLevelDomain function
    - deleted 4y ago
      [deleted]
thrdbndndn 4y ago
Why does the stack chart randomly change colors by just filtering/ordering?
- klausbreyer 4y ago
  I think plotly is giving the colors just based on the order of data inserted.
RomanPushkin 4y ago
> I had a database full of HN Stories since the very beginning, which accumulated to ~1GB. Just curious, is there any way to download that?
- klausbreyer 4y ago
  Sure, I can make a dump available. Can you tell a little bit, what you planning to do with it?
  - RomanPushkin 4y ago
    Well, I need titles. I had this idea for a while. Imagine bitcoin. At some point of time this word had emerged on hacker news, and we started seeing this more and more. I'm wondering what other words are there, that weren't in titles 2-3 years back? These words/topics are probably the future. If you ever know that, let me know by email (in my profile), or just post a comment here. There shouldn't be too much of them, probably hundreds.
    - klausbreyer 4y ago
      You've got mail :)
thomasjb 4y ago
Is there any information within .it, for the Italian provincial second level domains? I find the idea of these domains fascinating, although implementation of them could be problematic, given that countries can lose or gain territory over time (suppose the Kingdom of Naples secedes from Italy?)
Beefin 4y ago
here's what i've been using to automate domain name discovery: https://github.com/esteininger/domain-name-checker/blob/main/main.py https://github.com/esteininger/domain-name-checker/blob/main...
jamiemau 4y ago
Speaking of .io tld: Looks like the territory is going thru something (last paragraphs of https://en.wikipedia.org/wiki/.io#History https://en.wikipedia.org/wiki/.io#History mentions ".io domain could also be extinguished") Should domain owners be worried?
linux2647 4y ago
> Wrong links: http://blog.plover.com./prog/lib.html http://blog.plover.com./prog/lib.html That’s technically not incorrect. Host names, as far as DNS is concerned, always have a trailing “.” And my browser resolves the URL just fine
c7b 4y ago
One remark: .io looks like it's #3 in the time series, but it seems to be missing from bar plot.
- TonyTrapp 4y ago
  I see it in both graphs? Using Firefox here, in case it matters.
mthoms 4y ago
This is interesting, thanks. If someone wanted to dig deeper, does anyone know if Google makes the .dev zone file public?

68 points by klausbreyer 4y ago

This is great! It's something I've wondered about for a while. I was surprised to see the decline of .com is fairly linear. Before I looked I was expecting the use of alternate TLDs to be accelerating a bit. Are the absolute values the running totals? If so, why do they decline from 2021 to 2022? I think a graph for unique counts would be cool to see too. For example, the ClickHouse query posted earlier in this thread shows: domain count unique ------------------------- .org 349414 58226 .net 114499 31129 So the submissions using .org are 3x .net, but the unique domains seen using .org are less than 2x .net. I'm not sure if there's any significance there, but it would be interesting to see the difference. In the same context, I think it would be interesting to see the top 50 domains on each TLD. Anyway, it's very cool info to see. Thanks for sharing it!

klausbreyer 4y ago
Good feedback, thank you very much! I will see if I adress this in the future. :)

testernews 4y ago

domain.com./whatever is valid tho?

klausbreyer 4y ago
In theory, technically it is correct. But it was harder for me to parse and it really was not a significant amount.

gryn 4y ago

you don't need to scrape HN there's the public dataset in google BigQuery, I don't know if it's still updated regularly. edit: here's the link https://console.cloud.google.com/marketplace/details/y-combinator/hacker-news https://console.cloud.google.com/marketplace/details/y-combi... they seems to have stopped updating around the later part of 2022, don't know why.

pfarrell 4y ago
To anyone interested, I noticed this stoppage as well. IIRC, the data in BigQuery stopped being updated Nov 16, 2022. I’ve got a complete dataset that merges in new content from the firebase dataset, then rectifies story points at a later time (since stories can amass points indefinitely). Comment here if you’re interested. I’m thinking of publishing it as a torrent, but don’t know if there’s enough interest.
- malborodog 4y ago
  +1
- codetrotter 4y ago
  > Comment here if you’re interested. I’m thinking of publishing it as a torrent, but don’t know if there’s enough interest. Please do :)
  - gbertb 4y ago
    Would be interested as well!
- zxexz 4y ago
  I would love that! I find HN useful to have backed up on my laptop, for freetext searching offline mostly.
klausbreyer 4y ago
I also learned this today. Thanks for sharing! :) Edit: But sadly it is not up to date. :)

jossclimb 4y ago

I can't figure out what they did? Web scrapped hacker news for domain ideas?

skizm 4y ago

Dumb question: but I thought the ".io" TLD was owned by a really sketchy group / organization or ownership was being contested (or similar?) and having a domain there was semi-risky, is that still the case? I've avoided .io TLD because of this vague notion, but never really knew the specifics.

cybrexalpha 4y ago
If you're using .io because you want to use a "techy" TLD (that isn't .net that isn't held in high esteem for some reason), .dev is a great alternative.
- elcritch 4y ago
  Isn't that Google's TLD? Unfortunately I trust them less than the sketchy administrative situation of .io. Wake up one day and their algo's decided you're a spam account and lock you out and take your domain.
  - flangola7 4y ago
    .tech .technology .software .it .digital are options. I agree about avoiding Google.
    - klausbreyer 4y ago
      There is also .rest (for APIs ), .run and .codes ;)
  - cybrexalpha 4y ago
    You don't need to get it directly via Google. Mine are registered through a normal registrar, and not linked to a Google Account.
petercooper 4y ago
The biggest concern I've seen over the .io TLD is that it's the ccTLD for a territory ("British Indian Ocean Territory") whose existence / sovereignty is fragile. The people who originally lived in this territory were forcibly expelled in the 1970s – https://en.wikipedia.org/wiki/Expulsion_of_the_Chagossians https://en.wikipedia.org/wiki/Expulsion_of_the_Chagossians – so there are problems from two points of view: ethically this doesn't sit right with a lot of folks, but even from a pragmatic point of view, the UNGA adopted a resolution in 2019 specifying that the UK should withdraw its administration from the territory – this has not yet occurred, and it is uncertain what could/would happen with the ccTLD if or when it does.

ktpsns 4y ago

The interpretation you never asked for: The data exhibit a strong preferential attachment [1] behaviour, i.e. you can draw a line in a log-log plot (despite only a semilog plot is shown). This is typical for real world data. [1] https://en.wikipedia.org/wiki/Preferential_attachment https://en.wikipedia.org/wiki/Preferential_attachment

klausbreyer 4y ago
Thanks! A good thing to know!

zX41ZdbW 4y ago

This will save your time: https://play.clickhouse.com/play?user=play#U0VMRUNUIHRvcExldmVsRG9tYWluKHVybCkgQVMgZG9tYWluLCBjb3VudCgpIEFTIGMsIHN1bShzY29yZSksIHVuaXEoYnkpLCByb3VuZChhdmcoc2NvcmUpLCAzKSBGUk9NIGhhY2tlcm5ld3MgV0hFUkUgdHlwZSA9ICdzdG9yeScgQU5EIGRvbWFpbiAhPSAnJyBHUk9VUCBCWSAxIE9SREVSIEJZIDIgREVTQw== https://play.clickhouse.com/play?user=play#U0VMRUNUIHRvcExld... Takeaways: - .org is 50% better than .com; - .edu and .gov are really nice; - .io is cool, and much better than .uk; PS. I don't remember any rate limits of the API. Here is how I downloaded the data: https://github.com/ClickHouse/ClickHouse/issues/29693 https://github.com/ClickHouse/ClickHouse/issues/29693

tylerhannan 4y ago
Thanks zX41zdbW - I find the query you shared easier to reason about than: https://github.com/klausbreyer/hackernewsstats/blob/main/queries/count-years.sql https://github.com/klausbreyer/hackernewsstats/blob/main/que... Also: "Elapsed: 0.140 sec, read 33.95 million rows, 1.26 GB." 140 ms is quite impressive.
- return_to_monke 4y ago
  to be fair, that is large part because CH seems to have a topLevelDomain function
  - deleted 4y ago
    [deleted]

thrdbndndn 4y ago

Why does the stack chart randomly change colors by just filtering/ordering?

klausbreyer 4y ago
I think plotly is giving the colors just based on the order of data inserted.

RomanPushkin 4y ago

> I had a database full of HN Stories since the very beginning, which accumulated to ~1GB. Just curious, is there any way to download that?

klausbreyer 4y ago
Sure, I can make a dump available. Can you tell a little bit, what you planning to do with it?
- RomanPushkin 4y ago
  Well, I need titles. I had this idea for a while. Imagine bitcoin. At some point of time this word had emerged on hacker news, and we started seeing this more and more. I'm wondering what other words are there, that weren't in titles 2-3 years back? These words/topics are probably the future. If you ever know that, let me know by email (in my profile), or just post a comment here. There shouldn't be too much of them, probably hundreds.
  - klausbreyer 4y ago
    You've got mail :)

thomasjb 4y ago

Is there any information within .it, for the Italian provincial second level domains? I find the idea of these domains fascinating, although implementation of them could be problematic, given that countries can lose or gain territory over time (suppose the Kingdom of Naples secedes from Italy?)

Beefin 4y ago

here's what i've been using to automate domain name discovery: https://github.com/esteininger/domain-name-checker/blob/main/main.py https://github.com/esteininger/domain-name-checker/blob/main...

jamiemau 4y ago

Speaking of .io tld: Looks like the territory is going thru something (last paragraphs of https://en.wikipedia.org/wiki/.io#History https://en.wikipedia.org/wiki/.io#History mentions ".io domain could also be extinguished") Should domain owners be worried?

linux2647 4y ago

> Wrong links: http://blog.plover.com./prog/lib.html http://blog.plover.com./prog/lib.html That’s technically not incorrect. Host names, as far as DNS is concerned, always have a trailing “.” And my browser resolves the URL just fine

c7b 4y ago

One remark: .io looks like it's #3 in the time series, but it seems to be missing from bar plot.

TonyTrapp 4y ago
I see it in both graphs? Using Firefox here, in case it matters.

mthoms 4y ago

This is interesting, thanks. If someone wanted to dig deeper, does anyone know if Google makes the .dev zone file public?