3 ms·
So, you have a problem with Ahrefs, too? But what about Ezooms? That's a very annoying bot for my site, drives zero traffic. Also, what about Cyveillance? They
by bediger4000 13y ago
So, you have a problem with Ahrefs, too? But what about Ezooms? That's a very annoying bot for my site, drives zero traffic. Also, what about Cyveillance? They show up with a lying User-Agent field ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)"), but run Linux, and they ignore robots.txt by not even asking for it.
I actually blocked Ahrefs, Ezooms, Yandex and Cyveillance by IP address range in httpd.conf for a while, but I've decided to send them randomly-generated HTML (http://stratigery.com/bork.php http://stratigery.com/bork.php) instead. I'm really surprised by Ahrefs and Ezooms appetite for gibberish about naked celebrities, condiments and sweater puppies.
- devicenull 13y agoDoes Ahrefs not respect robots.txt? Their website says they do, https://ahrefs.com/robot/index.php https://ahrefs.com/robot/index.php
- bediger4000 13y agoI don't know if I tried that. Originally, I wanted to let them know they were forbidden, but now I just want to jerk them around.
- carsonreinke 13y agoBlock Yandex?
- bediger4000 13y agoI'd like to note that the above is the first time I've ever referenced that URL. No links exist to it on my web site. I just got a visit from the "MJ12Bot" (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php http://www.majestic12.co.uk/projects/dsearch/mj12bot.php) asking for robots.txt, and then the URL above. So, MJ12bot reads Hacker News, or at least follows as the URLs listed in it.