4 ms·

Mine is more awesome: http://logotype.se/robots.txt http://logotype.se/robots.txt

by logotype 13y ago

russellbeattie 13y ago
Heh, if only Yandex and Baidu respected robots.txt.
- agwa 13y ago
  I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker https://github.com/bluedragonz/bad-bot-blocker
  - bmelton 13y ago
    That's amazing. Know of anything like it for Nginx?
    - anon4 13y ago
      It's blocking by user agent and source ip. You should be able to port the list easily for Nginx. I'd even say you can write a simple awk script in a few minutes to convert from Apache's format to Nginx's.
    - coolj 13y ago
      curl -s https://raw.github.com/bluedragonz/bad-bot-blocker/master/.htaccess https://raw.github.com/bluedragonz/bad-bot-blocker/master/.h... | awk -F\" '/SetEnvIfNoCase/ {pattern = pattern $2 "|"} END {print "if ($http_user_agent ~ (" substr(pattern, 0, length(pattern)-1) ")) { return 403; }"}' Not tested. :)
  - anonymfus 13y ago
    >Unless your website is written in Russian or Chinese, you probably don't get any traffic from them. They mostly just waste bandwidth and consume resources. THIS is evil. You could use this argument for banning any new search engine.
    - ScottWhigham 13y ago
      What "new search engine" has actually generated actual revenue for any webmaster in the past ten years? You could argue DDG but that's the only one I can think of.
      - joveian 13y ago
        I think DDG primarily uses Yandex. They at least often put a "Powered by Yandex" logo along the side.
    - agwa 13y ago
      The problem is that both Yandex and Baidu are rather poorly behaved - they hit your website way too fast, downloading large bandwidth files in quick succession. That's actually what led me to the bad-bot-blocker project in the first place. Baidu has also been accused of not respecting robots.txt though I have not personally observed that. This is the reason they're blocked, not because they're new or non-English.
- kercker 13y ago
  I'm sure that Baidu respects robots.txt. This site coolshell.cn is boycotting baidu, so it tells Baiduspider that it doesn't want to be indexed by baidu. Baidu respects this and doesn't index anything from this website.
- anonymfus 13y ago
  Yandex's documentation about implemented robots.txt features: http://help.yandex.com/webmaster/controlling-robot/robots-txt.xml http://help.yandex.com/webmaster/controlling-robot/robots-tx... Yandex's online tool to check specific URL on allowness in robots.txt: http://webmaster.yandex.com/robots.xml http://webmaster.yandex.com/robots.xml
drdeadringer 13y ago
Is it analogue nuclear?
vladtaltos 13y ago
Why evil yandex?
anonymfus 13y ago
Why you consider Yandex evil?
- eksith 13y ago
  I don't know if this is universal, but Yandex was quite bad in the past at stumbling around generated links (E.G. calendars that go to infinity) and it wasn't at all unusual to see 400+ links crawled in a day or so. Baidu was the same, but I think they're more well behaved these days.
  - Moru 13y ago
    Lucky you with only 400+ per day. We had thousands per hour. Still the server survived but it wasn't fast...
- Cthulhu_ 13y ago
  I IP-banned Chinese search engine crawlers (yandex, baidu etc) because they did not respect the crawl-delay instruction in my robots.txt, causing my low-end crappy php server running vbulletin to bork out. Basically, their bots are too aggressive.
  - tbirdz 13y ago
    Yandex is Russian, not Chinese.
wfn 13y ago
Along similar lines, personally I was fascinated with https://bridges.torproject.org/robots.txt https://bridges.torproject.org/robots.txt :)
jonty 13y ago
Oh wow, I didn't realise my robots.txt joke had spread! I added that to the last.fm/robots.txt many years ago (http://www.wired.com/business/2010/08/robot-laws/all/ http://www.wired.com/business/2010/08/robot-laws/all/), and have just been made aware that it appears to have spread across the internet: https://www.google.co.uk/search?safe=off&q="Disallow%3A+%2Fignoring%2Fhuman%2Forders" https://www.google.co.uk/search?safe=off&q="Disallow%3A+%2Fi... Favourite sites I've found with it so far: php.net, princeton.edu, nest.com, songkick.com.

Mine is more awesome: http://logotype.se/robots.txt http://logotype.se/robots.txt

russellbeattie 13y ago

Heh, if only Yandex and Baidu respected robots.txt.

agwa 13y ago
I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker https://github.com/bluedragonz/bad-bot-blocker
- bmelton 13y ago
  That's amazing. Know of anything like it for Nginx?
  - anon4 13y ago
    It's blocking by user agent and source ip. You should be able to port the list easily for Nginx. I'd even say you can write a simple awk script in a few minutes to convert from Apache's format to Nginx's.
  - coolj 13y ago
    curl -s https://raw.github.com/bluedragonz/bad-bot-blocker/master/.htaccess https://raw.github.com/bluedragonz/bad-bot-blocker/master/.h... | awk -F\" '/SetEnvIfNoCase/ {pattern = pattern $2 "|"} END {print "if ($http_user_agent ~ (" substr(pattern, 0, length(pattern)-1) ")) { return 403; }"}' Not tested. :)
- anonymfus 13y ago
  >Unless your website is written in Russian or Chinese, you probably don't get any traffic from them. They mostly just waste bandwidth and consume resources. THIS is evil. You could use this argument for banning any new search engine.
  - ScottWhigham 13y ago
    What "new search engine" has actually generated actual revenue for any webmaster in the past ten years? You could argue DDG but that's the only one I can think of.
    - joveian 13y ago
      I think DDG primarily uses Yandex. They at least often put a "Powered by Yandex" logo along the side.
  - agwa 13y ago
    The problem is that both Yandex and Baidu are rather poorly behaved - they hit your website way too fast, downloading large bandwidth files in quick succession. That's actually what led me to the bad-bot-blocker project in the first place. Baidu has also been accused of not respecting robots.txt though I have not personally observed that. This is the reason they're blocked, not because they're new or non-English.
kercker 13y ago
I'm sure that Baidu respects robots.txt. This site coolshell.cn is boycotting baidu, so it tells Baiduspider that it doesn't want to be indexed by baidu. Baidu respects this and doesn't index anything from this website.
anonymfus 13y ago
Yandex's documentation about implemented robots.txt features: http://help.yandex.com/webmaster/controlling-robot/robots-txt.xml http://help.yandex.com/webmaster/controlling-robot/robots-tx... Yandex's online tool to check specific URL on allowness in robots.txt: http://webmaster.yandex.com/robots.xml http://webmaster.yandex.com/robots.xml

drdeadringer 13y ago

Is it analogue nuclear?

vladtaltos 13y ago

Why evil yandex?

anonymfus 13y ago

Why you consider Yandex evil?

eksith 13y ago
I don't know if this is universal, but Yandex was quite bad in the past at stumbling around generated links (E.G. calendars that go to infinity) and it wasn't at all unusual to see 400+ links crawled in a day or so. Baidu was the same, but I think they're more well behaved these days.
- Moru 13y ago
  Lucky you with only 400+ per day. We had thousands per hour. Still the server survived but it wasn't fast...
Cthulhu_ 13y ago
I IP-banned Chinese search engine crawlers (yandex, baidu etc) because they did not respect the crawl-delay instruction in my robots.txt, causing my low-end crappy php server running vbulletin to bork out. Basically, their bots are too aggressive.
- tbirdz 13y ago
  Yandex is Russian, not Chinese.

wfn 13y ago

Along similar lines, personally I was fascinated with https://bridges.torproject.org/robots.txt https://bridges.torproject.org/robots.txt :)

jonty 13y ago

Oh wow, I didn't realise my robots.txt joke had spread! I added that to the last.fm/robots.txt many years ago (http://www.wired.com/business/2010/08/robot-laws/all/ http://www.wired.com/business/2010/08/robot-laws/all/), and have just been made aware that it appears to have spread across the internet: https://www.google.co.uk/search?safe=off&q="Disallow%3A+%2Fignoring%2Fhuman%2Forders" https://www.google.co.uk/search?safe=off&q="Disallow%3A+%2Fi... Favourite sites I've found with it so far: php.net, princeton.edu, nest.com, songkick.com.