4 ms·
> Synthetic data has been estimated to be 5x more efficient than human data, you can make a model 1/5 size with similar performance. Interesting. Can you cite
by ShamelessC 3y ago
> Synthetic data has been estimated to be 5x more efficient than human data, you can make a model 1/5 size with similar performance.
Interesting. Can you cite anything to back up that claim?
- visarga 3y agoYes. > We follow the “Textbooks Are All You Need” approach, focusing this time on common sense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5, with performance on natural language tasks comparable to models 5x larger, and surpassing most non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic coding. https://arxiv.org/pdf/2309.05463.pdf https://arxiv.org/pdf/2309.05463.pdf
- ShamelessC 3y agoThanks!
- ignoramous 3y agoSee also Self-Play Fine-Tuning (SPIN) https://arxiv.org/abs/2401.01335 https://arxiv.org/abs/2401.01335 that (per authors) tunes as well as Direct Preference Optimization (DPO).