3 ms·

I have no trouble with Google as they have a way less aggressive crawl rate and Google actually sends me website traffic. Claude only takes. Google takes and g

• iLoveOncall 2y ago
  In the example above, Claude's crawler is on a whooping 0.0008 TPS, not exactly a DDOS attack. I'm not against blocking Claude for the reason you outline (or all LLM crawlers for that matter), but it's not exactly an "aggressive crawl rate".
  • spiderfarmer 2y ago
    On my server with 70 sites with about 100.000.000 unique pages I saw 60 urls per second regularly. Not even Googlebot hammers my server at that rate.
    • deleted 2y ago
      [deleted]
    • iLoveOncall 2y ago
      That's 0.00006% of all your pages per second. Even if it's sustained it really isn't much at all. At this rate sustained it would take more than 20 days to crawl all the pages. Again, this is really not painting the picture you want it to paint.