4 ms·

The fact that IA is so liberal with their scraping policies is laudable, because on the other end of the spectrum lies Wikimedia, where you're lucky if you mana

• shakesbeard 3y ago
  They provide monthly torrents. Can't imagine that to be too slow unless there's literally no one seeding.
• hgsgm 3y ago
  I just tried it. over 50MB in 1minute (6MB/hr). http://dumps.wikimedia.your.org/enwiki/20220820/ http://dumps.wikimedia.your.org/enwiki/20220820/ They also seed torrents
  • hgsgm 3y ago
    (er, 6MB/sec)
• Hello71 3y ago
  $ curl https://dumps.wikimedia.org/enwiki/20230520/enwiki-20230520-pages-articles-multistream.xml.bz2 -o/dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 2 20.4G 2 476M 0 0 4387k 0 1:21:38 0:01:51 1:19:47 4393k^C $ curl https://wikidata.aerotechnet.com/enwiki/20230520/enwiki-20230520-pages-articles-multistream.xml.bz2 -o/dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 20.4G 0 45.9M 0 0 1953k 0 3:03:23 0:00:24 3:02:59 2257k^C $ curl https://mirror.clarkson.edu/wikimedia/enwiki/20230520/enwiki-20230520-pages-articles-multistream.xml.bz2 -o/dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 1 20.4G 1 344M 0 0 35.2M 0 0:09:56 0:00:09 0:09:47 36.8M^C maybe you should check your own internet connection first?