3 ms·
What is the rationale here? Why is robots.txt not sufficient? Most search engines are already using AI, what separation are you trying to carve out with this qu
by verdverm 4y ago
What is the rationale here? Why is robots.txt not sufficient? Most search engines are already using AI, what separation are you trying to carve out with this question?
- rsolva 4y agoI would argue that there is a difference between: (1) indexing public content with the purpose of referring to the source in a search result and (2) ingesting content (no matter the license) to train models that can reproduce smaller or larger chunks later on. Google and others might be doing this with data from their crawlers already, but that is besides the point. I think these two uses of publicly available data is fundamentally different and that a distinction should be made.