4 ms·

Top level domains by popularity: grep -oP '\.[a-z]+(?=,)' current.csv | sort | uniq -c | sort -n ... 15840 .pl 17914 .it 20182 .de 21690 .in 2

by mg 4y ago

Top level domains by popularity: grep -oP '\.[a-z]+(?=,)' current.csv | sort | uniq -c | sort -n ... 15840 .pl 17914 .it 20182 .de 21690 .in 27812 .ru 29194 .jp 30359 .org 35741 .br 36675 .net 406052 .com .com domains by popularity: grep -oP '[a-z0-9-]+\.com(?=,)' current.csv | sort | uniq -c | sort -n ... 365 tistory.com 370 fc2.com 408 skipthegames.com 489 online.com 515 wordpress.com 707 uptodown.com 880 schoology.com 2570 fandom.com 2651 instructure.com 3244 blogspot.com

deleted 4y ago
[deleted]
voytec 4y ago
sort -r will reverse the order from most to less popular.
- layer8 4y ago
  .
  - codetrotter 4y ago
    uniq -c is in the pipeline because it counts the number of uniques
    - layer8 4y ago
      Ah, right, missed that.
egman_ekki 4y ago
Rather amazing seeing almost abandoned blogspot.com there at the top. Also interesting I haven’t heard about half of them. Some are nsfw, apparently.
- mometsi 4y ago
  These aren't sorted by number of visits, but by the number of rows in the list of most visited sites. Essentially which sites have the greatest number of frequently visited subdomains.
slim 4y ago
it's amazing that fandom is number 3 and wikipedia is not even there
- csande17 4y ago
  Wikipedia uses a .org domain, so it won't show up on "most popular .com domains" lists. (And I think the parent comment is searching for domains with lots of subdomains, which is why providers like Blogspot and Fandom show up.)
deleted 4y ago
[deleted]
kristianp 4y ago
Loading the data into the duckdb cli [0] and doing the first query: create table current as select * from '202211.csv'; select * from current; ┌────────────────────────────────────┬─────────┐ │ origin │ rank │ │ varchar │ int32 │ ├────────────────────────────────────┼─────────┤ │ https://hochi.news │ 1000 │ │ https://www.xnxx.xxx │ 1000 │ │ https://www.wordreference.com │ 1000 │ │ https://finance.naver.com │ 1000 │ │ https://www.macys.com │ 1000 │ │ https://www.xv-videos1.com │ 1000 │ │ https://fr.xhamster.com │ 1000 │ │ https://poki.com │ 1000 │ │ https://salonboard.com │ 1000 │ │ https://clgt.one │ 1000 │ select tld, count(*) from (select reverse(substr(reverse(origin),1, position('.' in reverse(origin))-1)) tld from current) group by tld order by count(*) desc; ┌───────────┬──────────────┐ │ tld │ count_star() │ │ varchar │ int64 │ ├───────────┼──────────────┤ │ com │ 406052 │ │ net │ 36675 │ │ br │ 35741 │ │ org │ 30359 │ │ jp │ 29194 │ │ ru │ 27812 │ │ in │ 21690 │ │ de │ 20182 │ │ it │ 17914 │ │ pl │ 15840 │ │ · │ · │ │ · │ · │ │ · │ · │ │ za:5002 │ 1 │ │ lk:8090 │ 1 │ │ org:1445 │ 1 │ │ co:14443 │ 1 │ │ ar:3016 │ 1 │ │ net:8001 │ 1 │ │ care:9624 │ 1 │ │ au:8443 │ 1 │ │ com:333 │ 1 │ │ edu:9016 │ 1 │ ├───────────┴──────────────┤ │ 2076 rows (20 shown) │ └──────────────────────────┘ [0] https://duckdb.org/docs/installation/ https://duckdb.org/docs/installation/
- Llamamoe 4y ago
  Are the bottom entries domain:port? I feel like the table would be more interesting excluding those.
azeemba 4y ago
It might be worth updating this comment and explaining your second query. People seem to think it is somehow measuring visits to those origins. But it's measuring how many unique subdomains are listed for those domains

by mg 4y ago

deleted 4y ago

[deleted]

voytec 4y ago

sort -r will reverse the order from most to less popular.

layer8 4y ago
.
- codetrotter 4y ago
  uniq -c is in the pipeline because it counts the number of uniques
  - layer8 4y ago
    Ah, right, missed that.

egman_ekki 4y ago

Rather amazing seeing almost abandoned blogspot.com there at the top. Also interesting I haven’t heard about half of them. Some are nsfw, apparently.

mometsi 4y ago
These aren't sorted by number of visits, but by the number of rows in the list of most visited sites. Essentially which sites have the greatest number of frequently visited subdomains.

slim 4y ago

it's amazing that fandom is number 3 and wikipedia is not even there

csande17 4y ago
Wikipedia uses a .org domain, so it won't show up on "most popular .com domains" lists. (And I think the parent comment is searching for domains with lots of subdomains, which is why providers like Blogspot and Fandom show up.)

deleted 4y ago

[deleted]

kristianp 4y ago

Loading the data into the duckdb cli [0] and doing the first query: create table current as select * from '202211.csv'; select * from current; ┌────────────────────────────────────┬─────────┐ │ origin │ rank │ │ varchar │ int32 │ ├────────────────────────────────────┼─────────┤ │ https://hochi.news │ 1000 │ │ https://www.xnxx.xxx │ 1000 │ │ https://www.wordreference.com │ 1000 │ │ https://finance.naver.com │ 1000 │ │ https://www.macys.com │ 1000 │ │ https://www.xv-videos1.com │ 1000 │ │ https://fr.xhamster.com │ 1000 │ │ https://poki.com │ 1000 │ │ https://salonboard.com │ 1000 │ │ https://clgt.one │ 1000 │ select tld, count(*) from (select reverse(substr(reverse(origin),1, position('.' in reverse(origin))-1)) tld from current) group by tld order by count(*) desc; ┌───────────┬──────────────┐ │ tld │ count_star() │ │ varchar │ int64 │ ├───────────┼──────────────┤ │ com │ 406052 │ │ net │ 36675 │ │ br │ 35741 │ │ org │ 30359 │ │ jp │ 29194 │ │ ru │ 27812 │ │ in │ 21690 │ │ de │ 20182 │ │ it │ 17914 │ │ pl │ 15840 │ │ · │ · │ │ · │ · │ │ · │ · │ │ za:5002 │ 1 │ │ lk:8090 │ 1 │ │ org:1445 │ 1 │ │ co:14443 │ 1 │ │ ar:3016 │ 1 │ │ net:8001 │ 1 │ │ care:9624 │ 1 │ │ au:8443 │ 1 │ │ com:333 │ 1 │ │ edu:9016 │ 1 │ ├───────────┴──────────────┤ │ 2076 rows (20 shown) │ └──────────────────────────┘ [0] https://duckdb.org/docs/installation/ https://duckdb.org/docs/installation/

Llamamoe 4y ago
Are the bottom entries domain:port? I feel like the table would be more interesting excluding those.

azeemba 4y ago

It might be worth updating this comment and explaining your second query. People seem to think it is somehow measuring visits to those origins. But it's measuring how many unique subdomains are listed for those domains