3 ms·
I don't think the site itself went down, so much as the scraping itself was excessive and beyond this, the presumed use of said scraping is against the TOS and
by tracker1 2y ago
I don't think the site itself went down, so much as the scraping itself was excessive and beyond this, the presumed use of said scraping is against the TOS and copyrights of the site itself.
On a smaller site, it can absolutely be brutal. I worked on a site where literally 2/3 of requests were bots and a lot of those were dynamically constructed pages that are/were harder to cache. In the end, switching to a supplemental search database helped significantly, but it doesn't mean that an aggressive bot is an okay thing.