3 ms·
But isn't that something that even smaller GPT-2 models demonstrated already?
by behnamoh 3y ago
But isn't that something that even smaller GPT-2 models demonstrated already?
- erichocean 3y agoNo, and this is covered in the various Phi papers, as well as TinyStories [0]: > Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. [0] https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759