3 ms·
I wonder if the reason for these results is that any data on the internet is already copied to other locations by actors who ignore crawling opt-outs. So, even
by stephen_cagle 1y ago
I wonder if the reason for these results is that any data on the internet is already copied to other locations by actors who ignore crawling opt-outs. So, even if they respect all web crawling opt-outs, they are still effectively copying the data because someone else did not respect it who does not include an opt-out.
- conradkay 1y agoMy guess is that it doesn't remove that much of the data, and the post-training data (not just randomly scraped from the web) probably matters more
- lllllm 1y agoYes this is an interesting question. In our arxiv paper [1] we did study this for news articles, and also removed duplicates of articles (decontamination). We did not observe an impact on the downstream accuracy of the LLM, in the case of news data. [1] https://arxiv.org/abs/2504.06219 https://arxiv.org/abs/2504.06219