9 ms·
Something like Common Crawl can be used for both search and AI training (and any other purpose, decided after crawl is done).
by zajio1am 1y ago
Something like Common Crawl can be used for both search and AI training (and any other purpose, decided after crawl is done).
- franga2000 1y agoThen such a crawler should mark itself with all purpose tags and thus be blocked in this scenario. Alternatively, it could make the request anyways and separate the crawled sites by permitted purpose in its output.