2 ms·
Hmm, perhaps robots.txt semantics needs to be extended to signify “OK to crawl for a search database but not as a training set”.
by gumby 4y ago
Hmm, perhaps robots.txt semantics needs to be extended to signify “OK to crawl for a search database but not as a training set”.