4 ms·
Couple other interesting papers from 2023 to add, specifically for small language models. TinyStories: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2
by dmezzetti 3y ago
Couple other interesting papers from 2023 to add, specifically for small language models.
TinyStories: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
Phi-1.5: https://arxiv.org/abs/2309.05463 https://arxiv.org/abs/2309.05463
Both these papers proved that with good/concise data we can get better performance with smaller parameter counts.
- mistrial9 3y agono I dont think so - TinyStories has a different architecture and assumptions for success.. different enough so that they dont compare to the others IMHO
- dmezzetti 3y agoPerhaps but TinyStories was one of the first papers to show that we don't need a lot of parameters to build a coherent generative model.
- mistrial9 3y agofwiw this link looks interesting, everyone https://github.com/neuml/txtai https://github.com/neuml/txtai
- p1esk 3y agoWhat is “coherent”?
- dmezzetti 3y agoGenerates plausible continuations of text.
- p1esk 3y agoBoth gpt-2 and gpt-4 do that.
- SgtBastard 3y ago> we don't need a lot of parameters to build a coherent generative model. GPT-4 is likely to be a trillion+ parameters over its MoR models.