6 ms·
Results are modest, maybe 20-30% fewer training steps to reach target performance. This won't solve the problem of organic data exhaustion. We need 100x more da
by visarga 1y ago
Results are modest, maybe 20-30% fewer training steps to reach target performance. This won't solve the problem of organic data exhaustion. We need 100x more data.
They didn't test against actual language model pretraining, only tested against a random init.
- A: Pre-trained on their synthetic LSTM data -> fine-tuned on Wikipedia
- B: Pre-trained on different natural language corpus -> fine-tuned on Wikipedia
- C: Random initialization -> fine-tuned on Wikipedia
They only test A vs C, not A vs B.
- WithinReason 1y agoThis paper addresses the problem of running out of data. You can't do B when you ran out of data so it's irrelevant.
- impossiblefork 1y ago20-30% isn't modest. I think there is a big problem though, but it's that it's character level prediction. It's not obvious how generate this kind of good synthetic data when it's to be fed to a tokenized model.