3 ms·
I just tried 2 http examples from https://commoncrawl.org/get-started https://commoncrawl.org/get-started for an old dataset and the most recent one and got 110
by ddorian43 3y ago
I just tried 2 http examples from https://commoncrawl.org/get-started https://commoncrawl.org/get-started for an old dataset and the most recent one and got 110Mb/s (my full download bandwidth):
wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz
wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2023-23/segments/1685224643388.45/warc/CC-MAIN-20230527223515-20230528013515-00483.warc.gz