3 ms·

IIRC LLMs also use common crawl data for training. Are they also blocking common crawl? Another thing is that chatgpt 4 can do live retrieval of websites in r

IIRC LLMs also use common crawl data for training. Are they also blocking common crawl? Another thing is that chatgpt 4 can do live retrieval of websites in response to users questions. That is a different crawler doing that I imagine. Are they going to block that too?

twelve40 3y ago
This. Unfortunately, there is common crawl, there is bing and a million of other ways they could hide/get the data from. Or, just ignore robots.txt, it's not like it's a very honest or transparent operation they run there.