3 ms·
If the output of this is even somewhat coherent, it would disprove the argument that mass amounts of copyrighted works are required to train an LLM. Unfortunate
by InvisibleUp 9mo ago
If the output of this is even somewhat coherent, it would disprove the argument that mass amounts of copyrighted works are required to train an LLM. Unfortunately that does not appear to be the case here.
- HighFreqAsuka 9mo agoTake a look at The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (https://arxiv.org/pdf/2506.05209 https://arxiv.org/pdf/2506.05209). They build a reasonable 7B parameter model using only open-licensed data.
- nickpsecurity 9mo agoThey mostly do that. They risked legal contamination by using Whisper-derived text and web text which might have gotchas. Other than that, it was a great collection for low-risk training.