6 ms·

I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker https://github.com/bluedragonz

by agwa 13y ago

I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker https://github.com/bluedragonz/bad-bot-blocker

bmelton 13y ago
That's amazing. Know of anything like it for Nginx?
- anon4 13y ago
  It's blocking by user agent and source ip. You should be able to port the list easily for Nginx. I'd even say you can write a simple awk script in a few minutes to convert from Apache's format to Nginx's.
- coolj 13y ago
  curl -s https://raw.github.com/bluedragonz/bad-bot-blocker/master/.htaccess https://raw.github.com/bluedragonz/bad-bot-blocker/master/.h... | awk -F\" '/SetEnvIfNoCase/ {pattern = pattern $2 "|"} END {print "if ($http_user_agent ~ (" substr(pattern, 0, length(pattern)-1) ")) { return 403; }"}' Not tested. :)
anonymfus 13y ago
>Unless your website is written in Russian or Chinese, you probably don't get any traffic from them. They mostly just waste bandwidth and consume resources. THIS is evil. You could use this argument for banning any new search engine.
- ScottWhigham 13y ago
  What "new search engine" has actually generated actual revenue for any webmaster in the past ten years? You could argue DDG but that's the only one I can think of.
  - joveian 13y ago
    I think DDG primarily uses Yandex. They at least often put a "Powered by Yandex" logo along the side.
- agwa 13y ago
  The problem is that both Yandex and Baidu are rather poorly behaved - they hit your website way too fast, downloading large bandwidth files in quick succession. That's actually what led me to the bad-bot-blocker project in the first place. Baidu has also been accused of not respecting robots.txt though I have not personally observed that. This is the reason they're blocked, not because they're new or non-English.

6 ms·

I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker https://github.com/bluedragonz

by agwa 13y ago

I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker https://github.com/bluedragonz/bad-bot-blocker

bmelton 13y ago
That's amazing. Know of anything like it for Nginx?
- anon4 13y ago
  It's blocking by user agent and source ip. You should be able to port the list easily for Nginx. I'd even say you can write a simple awk script in a few minutes to convert from Apache's format to Nginx's.
- coolj 13y ago
  curl -s https://raw.github.com/bluedragonz/bad-bot-blocker/master/.htaccess https://raw.github.com/bluedragonz/bad-bot-blocker/master/.h... | awk -F\" '/SetEnvIfNoCase/ {pattern = pattern $2 "|"} END {print "if ($http_user_agent ~ (" substr(pattern, 0, length(pattern)-1) ")) { return 403; }"}' Not tested. :)
anonymfus 13y ago
>Unless your website is written in Russian or Chinese, you probably don't get any traffic from them. They mostly just waste bandwidth and consume resources. THIS is evil. You could use this argument for banning any new search engine.
- ScottWhigham 13y ago
  What "new search engine" has actually generated actual revenue for any webmaster in the past ten years? You could argue DDG but that's the only one I can think of.
  - joveian 13y ago
    I think DDG primarily uses Yandex. They at least often put a "Powered by Yandex" logo along the side.
- agwa 13y ago
  The problem is that both Yandex and Baidu are rather poorly behaved - they hit your website way too fast, downloading large bandwidth files in quick succession. That's actually what led me to the bad-bot-blocker project in the first place. Baidu has also been accused of not respecting robots.txt though I have not personally observed that. This is the reason they're blocked, not because they're new or non-English.