4 ms·
Dumb question but just cuz I didn’t see it mentioned have you tried using a Disallow: / in your robots.txt? Or Crawl-delay: 10? That would be the first thing I
by n_u 1y ago
Dumb question but just cuz I didn’t see it mentioned have you tried using a Disallow: / in your robots.txt? Or Crawl-delay: 10? That would be the first thing I would try.
Sometimes these crawlers are just poorly written not malicious. Sometimes it’s both.
I would try a zip bomb next. I know there’s one that is 10 MB over the network and unzips to ~200TB.
- pknerd 1y agoIt's for crawlers not custom scrapers
- n_u 1y agoRespecting robots.txt is a convention not enforced by anything so yes the bot is certainly free to ignore it. But I’m not sure I understand your distinction. A scraper is a crawler regardless of whether it is “custom”or an off the shelf solution. The author also said the bot identifed itself as a crawler > Mozilla/5.0 (compatible; crawler)