4 ms·
"respecting web crawling opt-outs during data acquisition produces virtually no performance degradation" Great to read that!
by k__ 1y ago
"respecting web crawling opt-outs during data acquisition produces virtually no performance degradation"
Great to read that!
- JKCalhoun 1y agoIs there not yet a Source where the web has already been scraped and souped down to just the text? It would seem someone would have created such a thing in order to save LLM training from having to reinvent the wheel. I understand the web is a dynamic thing but still it would seem to be useful on some level.
- CaptainFever 1y agoCommon Crawl, maybe?
- stephen_cagle 1y agoI wonder if the reason for these results is that any data on the internet is already copied to other locations by actors who ignore crawling opt-outs. So, even if they respect all web crawling opt-outs, they are still effectively copying the data because someone else did not respect it who does not include an opt-out.
- conradkay 1y agoMy guess is that it doesn't remove that much of the data, and the post-training data (not just randomly scraped from the web) probably matters more
- lllllm 1y agoYes this is an interesting question. In our arxiv paper [1] we did study this for news articles, and also removed duplicates of articles (decontamination). We did not observe an impact on the downstream accuracy of the LLM, in the case of news data. [1] https://arxiv.org/abs/2504.06219 https://arxiv.org/abs/2504.06219
- Onavo 1y agoNo performance degradation on training metrics except for the end user. At the end of the day users and website owners have completely orthogonal interests. Users want answers and content, website owners want attention so they can upsell/push ads. You can only serve one master.
- esafak 1y ago> Users want answers and content, website owners want attention so they can upsell/push ads. You can only serve one master How are you going to serve users if web site owners decide to wall their content? You can't ignore one side of the market.
- Onavo 1y agoYou don't. You bypass them with crawlers and don't reveal your training data. And this is exactly why open source models can't surpass open weight models.
- diggan 1y ago> And this is exactly why open source models can't surpass open weight models. It is a fair point, but how strong of a point it is remains to be seen, some architectures are better than others, even with the same training data, so not impossible we could at one point see some innovative architectures beating current proprietary ones. It would probably be short-lived though, as the proprietary ones would obviously improve in their next release after that.