3 ms·
You could probably extract a lot from https://commoncrawl.org/ https://commoncrawl.org/
by onion2k 7mo ago
You could probably extract a lot from https://commoncrawl.org/ https://commoncrawl.org/
- ks2048 7mo agoHere's the latest (although, it looks truncated at those having > 1M pages), https://commoncrawl.github.io/cc-crawl-statistics/plots/tld/latestcrawl.html https://commoncrawl.github.io/cc-crawl-statistics/plots/tld/...
- ccgreg 7mo agoThe complete list hides in the web graph: https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2026-jan-feb-mar/index.html https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main... and the specific file that's every host we've seen in the latest 3 crawls is: https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2026-jan-feb-mar/host/cc-main-2026-jan-feb-mar-host-ranks.txt.gz https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main...