4 ms·
ArchiveTeam definitely do not intend to kill websites with too fast crawling, but definitely have done that unintentionally and always will stop/slow the crawli
by pabs3 7mo ago
ArchiveTeam definitely do not intend to kill websites with too fast crawling, but definitely have done that unintentionally and always will stop/slow the crawling when it happens.
Even the distributed crawling system has monitoring and controls to ensure it doesn't kill sites.