3 ms·
? I personally experienced many unbehaving crawlers/bots not respecting robots.txt and behaving like another user agent. Feel free to prove me wrong and disrup
by NicoJuicy 3y ago
? I personally experienced many unbehaving crawlers/bots not respecting robots.txt and behaving like another user agent.
Feel free to prove me wrong and disrupt cloudflare by only handling that use-case
- NicoJuicy 3y agoFound an old reference of mine what I did to block crawlers/bots https://news.ycombinator.com/item?id=34101988 https://news.ycombinator.com/item?id=34101988 > Had a lot of spammers with Russian language. Implemented expanding xml-bombs, Google Captcha, hidden input fields and a couple of other things against bots. But the block on the russian language was most effective ( and since I was dogfooding it, I didn't see the harm at the time. But it's out of scope at this very moment, yes).