4 ms·
I'm so tired of bots. A certain bot from Singapore started pulling all of the product images across multiple domains. Ok, whatever... Then we realized it never
by twothamendment 2y ago
I'm so tired of bots. A certain bot from Singapore started pulling all of the product images across multiple domains. Ok, whatever... Then we realized it never stopped. It made enough requests to download them 4-5x over and was still going. The AWS bill was not nice.
We added them to our robots.txt, but traffic didn't stop. I complained to their provider, who happened to also be AWS. Oh, the shock - they didn't care and recommended using robots.txt! AWS was making good money on both ends with this bot that apparently had more money to burn than we do.
- EGreg 2y agoThis is exactly why we need to move past “the Web” to precisely pay for storage and hosting with micropayments. It is the requester who should pay!
- sexy_seedbox 2y agoUse Cloudflare? Just give their server a captcha challenge?
- marcosdumay 2y agoBlock their origin network...
- quesera 2y agoI've been there. Can't you just 404 the bot in your reverse proxy, optionally setting logging to /dev/null? Still annoying, but a smaller AWS bill. And maybe the requester will eventually get the point.