3 ms·
Scrapping, especially on a large scale, can put a noticeable strain on servers. Bulk downloads (database dumps) are much cheaper to serve for someone crawling
by matkoniecz 5y ago
Scrapping, especially on a large scale, can put a noticeable strain on servers.
Bulk downloads (database dumps) are much cheaper to serve for someone crawling millions of pages.
It gets even more significant if generation of reply is resource intensive (not sure is Wikipedia qualifying for that but complex templates may cause this).