4 ms·

Claudebot is one of the most aggressive scraping bots I have encountered on my server in recent years. Because of that I'll block everything related to Claude.

JPLeRouzic 2y ago
For me (https://padiracinnovation.org https://padiracinnovation.org), it's Google which scans several times a day, which is astonishing for a blog updated once a week. ********** http://www.google.com/bot.html http://www.google.com/bot.html 449 http://www.bing.com/bingbot.htm http://www.bing.com/bingbot.htm 254 http://yandex.com/bots http://yandex.com/bots 159 imagesift.com 117 claudebot@anthropic.com 69 http://www.facebook.com/externalhit_uatext.php http://www.facebook.com/externalhit_uatext.php 36 https://dataforseo.com/dataforseo-bot https://dataforseo.com/dataforseo-bot 34 http://www.apple.com/go/applebot http://www.apple.com/go/applebot 32 https://webmaster.petalsearch.com/site/petalbot https://webmaster.petalsearch.com/site/petalbot 22 https://scrapy.org https://scrapy.org 9 http://www.semrush.com/bot.html http://www.semrush.com/bot.html 4 https://developers.google.com/ https://developers.google.com/ 2 https://openai.com/bot https://openai.com/bot" 2 http://www.bing.com/bingbot.htm http://www.bing.com/bingbot.htm 2 http://code.google.com/appengine http://code.google.com/appengine; appid: s~feedly-nikon3 2 http://www.pinterest.com/bot.html http://www.pinterest.com/bot.html 1 https://www.mojeek.com/bot.html https://www.mojeek.com/bot.html 1 http://www.reddit.com/feedback http://www.reddit.com/feedback 1 https://github.com/adbar/trafilatura https://github.com/adbar/trafilatura 1
- spiderfarmer 2y ago
  I have no trouble with Google as they have a way less aggressive crawl rate and Google actually sends me website traffic. Claude only takes. Google takes and gives.
  - iLoveOncall 2y ago
    In the example above, Claude's crawler is on a whooping 0.0008 TPS, not exactly a DDOS attack. I'm not against blocking Claude for the reason you outline (or all LLM crawlers for that matter), but it's not exactly an "aggressive crawl rate".
    - spiderfarmer 2y ago
      On my server with 70 sites with about 100.000.000 unique pages I saw 60 urls per second regularly. Not even Googlebot hammers my server at that rate.
      - deleted 2y ago
        [deleted]
      - iLoveOncall 2y ago
        That's 0.00006% of all your pages per second. Even if it's sustained it really isn't much at all. At this rate sustained it would take more than 20 days to crawl all the pages. Again, this is really not painting the picture you want it to paint.
- rurban 2y ago
  For me it's semrush, not being able to understand robots.txt wildcards, and 20x more traffic than normal bots. What are they doing on a movie site anyway? No problem with claudebot

JPLeRouzic 2y ago

For me (https://padiracinnovation.org https://padiracinnovation.org), it's Google which scans several times a day, which is astonishing for a blog updated once a week. ********** http://www.google.com/bot.html http://www.google.com/bot.html 449 http://www.bing.com/bingbot.htm http://www.bing.com/bingbot.htm 254 http://yandex.com/bots http://yandex.com/bots 159 imagesift.com 117 claudebot@anthropic.com 69 http://www.facebook.com/externalhit_uatext.php http://www.facebook.com/externalhit_uatext.php 36 https://dataforseo.com/dataforseo-bot https://dataforseo.com/dataforseo-bot 34 http://www.apple.com/go/applebot http://www.apple.com/go/applebot 32 https://webmaster.petalsearch.com/site/petalbot https://webmaster.petalsearch.com/site/petalbot 22 https://scrapy.org https://scrapy.org 9 http://www.semrush.com/bot.html http://www.semrush.com/bot.html 4 https://developers.google.com/ https://developers.google.com/ 2 https://openai.com/bot https://openai.com/bot" 2 http://www.bing.com/bingbot.htm http://www.bing.com/bingbot.htm 2 http://code.google.com/appengine http://code.google.com/appengine; appid: s~feedly-nikon3 2 http://www.pinterest.com/bot.html http://www.pinterest.com/bot.html 1 https://www.mojeek.com/bot.html https://www.mojeek.com/bot.html 1 http://www.reddit.com/feedback http://www.reddit.com/feedback 1 https://github.com/adbar/trafilatura https://github.com/adbar/trafilatura 1

spiderfarmer 2y ago
I have no trouble with Google as they have a way less aggressive crawl rate and Google actually sends me website traffic. Claude only takes. Google takes and gives.
- iLoveOncall 2y ago
  In the example above, Claude's crawler is on a whooping 0.0008 TPS, not exactly a DDOS attack. I'm not against blocking Claude for the reason you outline (or all LLM crawlers for that matter), but it's not exactly an "aggressive crawl rate".
  - spiderfarmer 2y ago
    On my server with 70 sites with about 100.000.000 unique pages I saw 60 urls per second regularly. Not even Googlebot hammers my server at that rate.
    - deleted 2y ago
      [deleted]
    - iLoveOncall 2y ago
      That's 0.00006% of all your pages per second. Even if it's sustained it really isn't much at all. At this rate sustained it would take more than 20 days to crawl all the pages. Again, this is really not painting the picture you want it to paint.
rurban 2y ago
For me it's semrush, not being able to understand robots.txt wildcards, and 20x more traffic than normal bots. What are they doing on a movie site anyway? No problem with claudebot