3 ms·

> Does anyone know what's the deal with these scrapers, or why they're attributed to AI? You don't really need to guess, it's obvious from the access logs. I r

by Tharre 8mo ago

> Does anyone know what's the deal with these scrapers, or why they're attributed to AI? You don't really need to guess, it's obvious from the access logs. I realize not everyone runs their own server, so here are a couple excerpts from mine to illustrate: - "meta-externalagent/1.1 +https://developers.facebook.com/docs/sharing/webmasters/crawler https://developers.facebook.com/docs/sharing/webmasters/craw...)" - "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" - "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36" - "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot https://openai.com/gptbot)" - [...] (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot https://webmaster.petalsearch.com/site/petalbot)" And to give a sense of scale, my cgit instance recieved 37 212 377 requests over the last 60 days, >99% of which are bots. The access.log from nginx grew to 12 GiB in those 60 days. They scrape everything they can find, indiscriminately, including endpoints that have to do quite a bit of work, leading to a baseline 30-50% CPU utilization on that server right now. Oh, and of course, almost nothing of what they are scraping actually changed in the last 60 days, it's literally just a pointless waste of compute and bandwidth. I'm actually surprised that the hosting companies haven't blocked all of them yet, this has to increase their energy bills substantially. Some bots also seem better behaved then others, OpenAI alone accounts for 26 million of those 37 million requests.

everybodyknows 8mo ago
Following your link above, https://openai.com/gptbot https://openai.com/gptbot > ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. So, not AI training in this case, nor any other large-batch scraping, but rather inference-time Retrieval Augmented Generation, with the "retrieval" happening over the web?
- groby_b 8mo ago
  Likely, at least for some. I've caught various chatbots/CLI harnesses more than once inspecting a github repo file by file (often multiple times, because context rot) But the sheer volume makes it unlikely that's the only reason. It's not like everybody has constantly questions bout the same tiny website.
- Tharre 8mo ago
  Those would have the user agent "ChatGPT-User" though, and I barely see those. The majority comes from "GPTBot" like in my excerpt above, which makes it pretty clear that it's used for some sort of training: "GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models."

by Tharre 8mo ago

everybodyknows 8mo ago

Following your link above, https://openai.com/gptbot https://openai.com/gptbot > ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. So, not AI training in this case, nor any other large-batch scraping, but rather inference-time Retrieval Augmented Generation, with the "retrieval" happening over the web?

groby_b 8mo ago
Likely, at least for some. I've caught various chatbots/CLI harnesses more than once inspecting a github repo file by file (often multiple times, because context rot) But the sheer volume makes it unlikely that's the only reason. It's not like everybody has constantly questions bout the same tiny website.
Tharre 8mo ago
Those would have the user agent "ChatGPT-User" though, and I barely see those. The majority comes from "GPTBot" like in my excerpt above, which makes it pretty clear that it's used for some sort of training: "GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models."