4 ms·
In the example above, Claude's crawler is on a whooping 0.0008 TPS, not exactly a DDOS attack. I'm not against blocking Claude for the reason you outline (or a
by iLoveOncall 2y ago
In the example above, Claude's crawler is on a whooping 0.0008 TPS, not exactly a DDOS attack.
I'm not against blocking Claude for the reason you outline (or all LLM crawlers for that matter), but it's not exactly an "aggressive crawl rate".
- spiderfarmer 2y agoOn my server with 70 sites with about 100.000.000 unique pages I saw 60 urls per second regularly. Not even Googlebot hammers my server at that rate.
- deleted 2y ago[deleted]
- iLoveOncall 2y agoThat's 0.00006% of all your pages per second. Even if it's sustained it really isn't much at all. At this rate sustained it would take more than 20 days to crawl all the pages. Again, this is really not painting the picture you want it to paint.