3 ms·
Are you aware of any efforts to do this? Even a 3B param attempt would be informative.
by hedgehog 2y ago
Are you aware of any efforts to do this? Even a 3B param attempt would be informative.
- nickpsecurity 2y agoHere is the only legal efforts I know about that’s available in some way: https://www.fairlytrained.org/ https://www.fairlytrained.org/ https://www.kl3m.ai/#features https://www.kl3m.ai/#features Here’s a dataset that could be used for a public domain model: https://www.tensorflow.org/datasets/catalog/pg19 https://www.tensorflow.org/datasets/catalog/pg19 If non-public domain, one can add in the code from The Stack. That would be tens of gigabytes of both English text and code. Then, third-party could add licensed, modern works to the model with further pre-training. I also think a model trained on a large amount of public domain data would be good for experimentation with reproduceability. There would be no intellectual property issues in the reproduction of the results. Should also be useful in a lot of ways.