5 ms·
Ask HN: Will we see a license explicitly excluding AIs from scraping content?
In addition, should we make an equivalent to robots.txt or include some AI-related rules into robots.txt?
- verdverm 4y agoWhat is the rationale here? Why is robots.txt not sufficient? Most search engines are already using AI, what separation are you trying to carve out with this question?
- rsolva 4y agoI would argue that there is a difference between: (1) indexing public content with the purpose of referring to the source in a search result and (2) ingesting content (no matter the license) to train models that can reproduce smaller or larger chunks later on. Google and others might be doing this with data from their crawlers already, but that is besides the point. I think these two uses of publicly available data is fundamentally different and that a distinction should be made.
- buro9 4y agoInterestingly Google have a big advantage here, almost everyone wants to be top-ranked on Google and will gladly allow Google to crawl and scrape even when they aggressively close the door on everyone else crawling and scraping. Google won't separate these things out, you get ranked high in the search engine _and_ your content goes into the AI machine, or you get neither.