3 ms·
There is common crawl: https://github.com/commoncrawl https://github.com/commoncrawl
by ddorian43 3y ago
There is common crawl: https://github.com/commoncrawl https://github.com/commoncrawl
- marginalia_nu 3y agoKinda sucks that it's stuck in AWS with no easy way of exfiltrating the data from the Amazon ecosystem. Last I tried I got like 100 Kb/s on their HTTP mirror. At that rate, the download would take 12 years.
- ddorian43 3y agoI just tried 2 http examples from https://commoncrawl.org/get-started https://commoncrawl.org/get-started for an old dataset and the most recent one and got 110Mb/s (my full download bandwidth): wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2023-23/segments/1685224643388.45/warc/CC-MAIN-20230527223515-20230528013515-00483.warc.gz