3 ms·
IMO, if the AI were more sample-efficient (a long-standing problem that predates LLMs), they would be able to learn from purely open-licensed content, which I t
by ben_w 2y ago
IMO, if the AI were more sample-efficient (a long-standing problem that predates LLMs), they would be able to learn from purely open-licensed content, which I think Wikipedia (CC-BY-SA) would be an example of? I think they'd even pass the share-alike requirements, given Meta are giving away the model weights?
https://en.wikipedia.org/wiki/Wikipedia:Copyrights https://en.wikipedia.org/wiki/Wikipedia:Copyrights
- wizzwizz4 2y agoSince this is Wikipedia, it could even satisfy the attribution requirements (though most CC-licensed corpora require attributing the individual authors).
- visarga 2y agoAlteratively if they trained the model on synthetic data, filtered to avoid duplication, then no copyrighted material would be seen by the model. For example turn an article into QA pairs, or summarize across multiple sources of text.
- thesz 2y ago> trained the model on synthetic data You get knowledge collapse [1] this way. [1] https://arxiv.org/abs/2404.03502 https://arxiv.org/abs/2404.03502
- ben_w 2y agoYou can get knowledge collapse. But even though there are counter-examples (e.g. learning Go from self-play based only on the rules), IMO — based on how often people were already doing this with buggy human-written software[0][1] — most people don't think about this in the right way and will therefore treat these things as magical oracles when they shouldn't. No silver bullets. [0] https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal [1] https://en.wikipedia.org/wiki/Computer_says_no https://en.wikipedia.org/wiki/Computer_says_no