3 ms·
If a bot ignores robots.txt that's a paddlin'. Right to the blacklist.
by jandrese 2y ago
If a bot ignores robots.txt that's a paddlin'. Right to the blacklist.
- deleted 2y ago[deleted]
- nabla9 2y agoThe linked article explains what happens when you block their IP.
- gs17 2y agoFor reference: > If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). It's really absurd that they seem to think this is acceptable.
- viraptor 2y agoBlock the whole ASN in that case.
- therealdrag0 2y agoWhat about adding fake sleeps?