4 ms·

There are really 3 axes - the number of tokens, the number of parameters, and the number of flops per pass.

• p1esk 3y ago
  This says nothing about data quality.
  • Salgat 3y ago
    To add, if you have a model that's too large for the dataset, you end up in a situation where it just memorizes the training dataset and performs poorly on the test data. That's why GPT works so good, you only pass the training data through it once, and there's so much good training data that it works in spite of the model being absolutely massive.
    • whimsicalism 3y ago
      it only memorizes the dataset if you give it multiple passes and are bad at regularizing overparameterization is fine