4 ms·
As a network operator that runs a number of websites I'm honestly one step away from null routing all of their traffic. I'm constantly seeing their bots and as
by icehawk 4y ago
As a network operator that runs a number of websites I'm honestly one step away from null routing all of their traffic. I'm constantly seeing their bots and as far as I know it's not helping anything.
- bpicolo 4y agoThey represented something like 30% of traffic at a couple places I worked. Their bots are not well behaved
- dazc 4y agoWhat's stopping you, they crawl your site day and night and sell the data they have harvested at a vast profit. They don't even respect robots.txt.
- botsbreeder 4y agoAhrefsBot respects robots.txt. Details are here: https://ahrefs.com/robot https://ahrefs.com/robot
- blitzar 4y agoMy parents say I am a very good boy too.
- agmater 4y agoIf you don't believe the CEO you can always just IP block them. In my experience they respect "User-agent: * Disallow: /".
- dazc 4y agoBlocking the user agent in my WAF is the only thing that actually worked. So, I'm sure you're right and I am just a random outlying case.