4 ms·

Ahrefs could probably save another few hundred million if they didn't repeatedly visit the same links over and over indefinitely to find the same error codes or

Ahrefs could probably save another few hundred million if they didn't repeatedly visit the same links over and over indefinitely to find the same error codes or a binary file that they surely don't care about. I see them in my logs for my podcast hosting service, hitting the same 404s for months and months. They end up hitting audio files and downloading (or attempting to download) many gigabytes of content each day. They don't do anything with audio! As soon as they get the HTTP headers, they could say "oops, I don't care about this" and disconnect. And in many cases, they got those URLs from the enclosure tags of RSS feeds... I'm not sure what they were expecting! In the cloud, not in the cloud, frankly it seems to me like their business would be far more efficient if they tuned what their crawler actually crawled.

lazyfanatic 4y ago
No, throw hardware at it! My life for the last 25 years has been trying to optimize code via I/O. My favorite design pattern is a for loop that tries once per item and completely fails to never try again, nor report it failed at all. /s
drcongo 4y ago
Ahrefs is like my white whale. I hate them. I can't stop myself from battling them. I hate the fact that they ignore status codes, I hate the fact that they ignore crawl rate specs in robots.txt, I hate the fact that they crawl URLs on my sites that have never and will never exist and keep coming back to do it again.
- nazka 4y ago
  Why not use Semrush?
  - paraboul 4y ago
    You don't have to "use" ahrefs for them to crawl your website
  - drcongo 4y ago
    I don't use ahrefs or semrush, and in fact, semrush's crawlers behave even worse.
- phendrenad2 4y ago
  Why not just block any requests with Ahrefs user-agent?
hinkley 4y ago
I cannot express emphatically enough how powerful the concept of classes of service can be for simplifying your production issues. I think people see the notion of a network neutrality as an all or nothing thing. Network Neutrality between networks, yes, but within your racks you’re trying to balance a large equation instead of two or three smaller ones. If they reclassified sketchy pages they wouldn’t have nearly as much of the sorts of trouble you mention.

lazyfanatic 4y ago

No, throw hardware at it! My life for the last 25 years has been trying to optimize code via I/O. My favorite design pattern is a for loop that tries once per item and completely fails to never try again, nor report it failed at all. /s

drcongo 4y ago

Ahrefs is like my white whale. I hate them. I can't stop myself from battling them. I hate the fact that they ignore status codes, I hate the fact that they ignore crawl rate specs in robots.txt, I hate the fact that they crawl URLs on my sites that have never and will never exist and keep coming back to do it again.

nazka 4y ago
Why not use Semrush?
- paraboul 4y ago
  You don't have to "use" ahrefs for them to crawl your website
- drcongo 4y ago
  I don't use ahrefs or semrush, and in fact, semrush's crawlers behave even worse.
phendrenad2 4y ago
Why not just block any requests with Ahrefs user-agent?

hinkley 4y ago

I cannot express emphatically enough how powerful the concept of classes of service can be for simplifying your production issues. I think people see the notion of a network neutrality as an all or nothing thing. Network Neutrality between networks, yes, but within your racks you’re trying to balance a large equation instead of two or three smaller ones. If they reclassified sketchy pages they wouldn’t have nearly as much of the sorts of trouble you mention.