> they used common crawl which respects individual site policies such as robots.txt Nice, so what? I seriously doubt that authors of the content crawled by Com

> they used common crawl which respects individual site policies such as robots.txt Nice, so what? I seriously doubt that authors of the content crawled by CommonCrawl agreed to terms that their content will be used by openai. Moreover CC seems to be opt-in by default according to their faq: > You configure your robots.txt file which uses the Robots Exclusion Protocol to block the crawler. Our bot’s Exclusion User-Agent string is: CCBot. Add these lines to your robots.txt file and our crawler will stop crawling your website Again, I doubt that plenty of people are putting CC specific rules into their robots.txt, moreover I'm not naive and I doubt that in our reality where "move fast and break things" is THE motto for any big corpo any rules like that are respected at all.