4 ms·
Source: a bad study from 2023.
by FeepingCreature 7mo ago
Source: a bad study from 2023.
- slowmovintarget 7mo agoWhy is the study bad? https://www.nature.com/articles/s41586-024-07566-y https://www.nature.com/articles/s41586-024-07566-y
- FeepingCreature 7mo agoBecause they exclusively used a model that was about as big as the original GPT-2. Which, I mean, fair enough within these constraints, but it's cited like it's a universal law. Really all that can be taken away from the study is "we trained a very small model on data generated from it in a particular way, and this was eventually harmful for the model." Also note that models are nowadays trained on massively self-generated data (task RL post-training) and it seems to significantly improve their performance.