3 ms·
Curious that they chose to use GPT-2-XL, given the age of that model. I guess they wanted a small model (1.5B) and started work on this quite a while ago. Today
by wongarsu 2y ago
Curious that they chose to use GPT-2-XL, given the age of that model. I guess they wanted a small model (1.5B) and started work on this quite a while ago. Today there is a decent selection of much more capable 1.5B models (Quen2-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, TinySwallow 1.5B, Stella 1.5B, Qwen2.5-Math-1.5B, etc). But they are all derived from the Qwen series of models, which wasn't available when they started this research.
- littlestymaar 2y agoTinyLlama would have worked too and is older than the Qwen family.
- nickpsecurity 2y agoI was collecting examples of models trained on single GPU or with very low cost. A number of projects used Bert or GPT to since the implementations were very simple with with some components optimized. There’s also a lot of projects that have trained Bert in GPT two models which make for more scientific comparisons. With no other information, those would be my guesses as to why one would use a GPT2 model.
- deleted 2y ago[deleted]
- ImHereToVote 2y agoI understand GPT-2 has been somewhat mapped to a certain extent.
- scoresmoke 2y agoGPT-2 follows the very well-studied architecture of Transformer decoder, so the outcomes of this study might be applicable to the more complicated models.
- imjonse 2y agoThe paper predates Qwen2 and R1, this work is probably a year old.
- Sharlin 2y agoYou can think of GPT-2 as the D. melanogaster of language models.