4 ms·
Maybe something like an extension of robots.txt. E.g. if your don't allow crawling then the data is not public. I think that's fair, after all if you want compl
by tensor 3y ago
Maybe something like an extension of robots.txt. E.g. if your don't allow crawling then the data is not public. I think that's fair, after all if you want complete strangers to be able to search for it arguably you are asking for it to be public.
- JohnFen 3y agoThe problem with robots.txt is that it depends on the crawler to respect it (and many do not). It's not an adequate protection against crawlers in general and a specific form just for AI wouldn't be adequate either, for the same reason.