49 ms·

The Chinchilla laws were compute optimal scaling laws. They're not supposed to tell you what parameter-token combination will saturate a model.

moffkalast 3y ago
Compute optimal for what, training? There's nothing optimal in blowing up model size beyond the absolute minimal needed or you'll spent the equivalent of a country in electricity trying to scale inference later.
- rfw300 3y ago
  Yes, compute-optimal for training only. The purpose of the paper wasn’t to determine the most economically practical model one could build, the most “intelligent” model one could build given some amount of compute.
  - ijk 3y ago
    Quite. The big question at the time was "how much data do we need to train GPT-3 equivalent models". Open models had failed to live up to GPT performance, even ones with a massive number of parameters. So getting results that suggested a reason why other models were massively undertrained was important. Meanwhile, people noticed that for deployed models, inference cost often outweighs the initial training costs. It's sometimes better to train a smaller, faster model longer on more data, because it has lower overall cost (including environmental impact) if you're expecting to run the model a few million or billion times (e.g., [1]). So training past the Chinchilla optimum point became a lot more common, particularly after Llama. [1] https://arxiv.org/abs/2401.00448 https://arxiv.org/abs/2401.00448
- FeepingCreature 3y ago
  Blow up model size, get lots of space and parameters to do the double-descent grok thing in, then distill it way way down?
- famouswaffles 3y ago
  Training yes. Doubling your parameter count past that ratio will yield a better model than doubling your data and is much easier and cheaper to do.
  - naasking 3y ago
    That suggests that it's likely memorizing more special cases rather than distilling general principles. They generalize to some degree but clearly there's room for improvement.
    - famouswaffles 3y ago
      It doesn't really suggest anything. Neither model will even close to saturation and all else equal, bigger models perform better in every way, including generalization.
      - naasking 3y ago
        But why do bigger models perform better? Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing.
        famouswaffles 3y ago
        >But why do bigger models perform better? No one really knows the answer to this question. >Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing. What I'm trying to say is that both models in either scenario are very over-parameterized and under-trained. You say the answer is extra space ? The smaller model has not used anywhere near the space it has. They both have extra space. It's like arguing a bigger drum is better because of extra space when all the water you plan to store will not take even half of your smaller drum

moffkalast 3y ago

Compute optimal for what, training? There's nothing optimal in blowing up model size beyond the absolute minimal needed or you'll spent the equivalent of a country in electricity trying to scale inference later.

rfw300 3y ago
Yes, compute-optimal for training only. The purpose of the paper wasn’t to determine the most economically practical model one could build, the most “intelligent” model one could build given some amount of compute.
- ijk 3y ago
  Quite. The big question at the time was "how much data do we need to train GPT-3 equivalent models". Open models had failed to live up to GPT performance, even ones with a massive number of parameters. So getting results that suggested a reason why other models were massively undertrained was important. Meanwhile, people noticed that for deployed models, inference cost often outweighs the initial training costs. It's sometimes better to train a smaller, faster model longer on more data, because it has lower overall cost (including environmental impact) if you're expecting to run the model a few million or billion times (e.g., [1]). So training past the Chinchilla optimum point became a lot more common, particularly after Llama. [1] https://arxiv.org/abs/2401.00448 https://arxiv.org/abs/2401.00448
FeepingCreature 3y ago
Blow up model size, get lots of space and parameters to do the double-descent grok thing in, then distill it way way down?
famouswaffles 3y ago
Training yes. Doubling your parameter count past that ratio will yield a better model than doubling your data and is much easier and cheaper to do.
- naasking 3y ago
  That suggests that it's likely memorizing more special cases rather than distilling general principles. They generalize to some degree but clearly there's room for improvement.
  - famouswaffles 3y ago
    It doesn't really suggest anything. Neither model will even close to saturation and all else equal, bigger models perform better in every way, including generalization.
    - naasking 3y ago
      But why do bigger models perform better? Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing.
      - famouswaffles 3y ago
        >But why do bigger models perform better? No one really knows the answer to this question. >Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing. What I'm trying to say is that both models in either scenario are very over-parameterized and under-trained. You say the answer is extra space ? The smaller model has not used anywhere near the space it has. They both have extra space. It's like arguing a bigger drum is better because of extra space when all the water you plan to store will not take even half of your smaller drum