3 ms·

Key claims: "We have found three potential issues with Hoffmann et al.’s estimates of the Chinchilla scaling law that rely on Approach 3: 1. Their estimated mo

Key claims: "We have found three potential issues with Hoffmann et al.’s estimates of the Chinchilla scaling law that rely on Approach 3: 1. Their estimated model fits the reconstructed data very poorly. These conclusions hold even when accounting for potential noise in data reconstruction and excluding outlier models. 2. The confidence are implausibly tight given the number of data points. Obtaining confidence intervals that tight would require many hundreds of thousands of observations, while they likely had only ∼400. 3. Their estimated model implies a scaling policy that is inconsistent with their other approach" Data point most people are probably looking for: "We find a range consistent with the 20 tokens per parameter rule of thumb. Indeed, our point estimates imply that 25.6 tokens per parameters is optimal."

moffkalast 2y ago
Their rule of thumb would imply that a 70B model is saturated with 1.7T tokens, that's inconsistent with reality.
- famouswaffles 2y ago
  The Chinchilla laws were compute optimal scaling laws. They're not supposed to tell you what parameter-token combination will saturate a model.
  - moffkalast 2y ago
    Compute optimal for what, training? There's nothing optimal in blowing up model size beyond the absolute minimal needed or you'll spent the equivalent of a country in electricity trying to scale inference later.
    - rfw300 2y ago
      Yes, compute-optimal for training only. The purpose of the paper wasn’t to determine the most economically practical model one could build, the most “intelligent” model one could build given some amount of compute.
      - ijk 2y ago
        Quite. The big question at the time was "how much data do we need to train GPT-3 equivalent models". Open models had failed to live up to GPT performance, even ones with a massive number of parameters. So getting results that suggested a reason why other models were massively undertrained was important. Meanwhile, people noticed that for deployed models, inference cost often outweighs the initial training costs. It's sometimes better to train a smaller, faster model longer on more data, because it has lower overall cost (including environmental impact) if you're expecting to run the model a few million or billion times (e.g., [1]). So training past the Chinchilla optimum point became a lot more common, particularly after Llama. [1] https://arxiv.org/abs/2401.00448 https://arxiv.org/abs/2401.00448
    - FeepingCreature 2y ago
      Blow up model size, get lots of space and parameters to do the double-descent grok thing in, then distill it way way down?
    - famouswaffles 2y ago
      Training yes. Doubling your parameter count past that ratio will yield a better model than doubling your data and is much easier and cheaper to do.
      - naasking 2y ago
        That suggests that it's likely memorizing more special cases rather than distilling general principles. They generalize to some degree but clearly there's room for improvement.
        famouswaffles 2y ago
        It doesn't really suggest anything. Neither model will even close to saturation and all else equal, bigger models perform better in every way, including generalization.
        naasking 2y ago
        But why do bigger models perform better? Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing.
        famouswaffles 2y ago
        >But why do bigger models perform better? No one really knows the answer to this question. >Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing. What I'm trying to say is that both models in either scenario are very over-parameterized and under-trained. You say the answer is extra space ? The smaller model has not used anywhere near the space it has. They both have extra space. It's like arguing a bigger drum is better because of extra space when all the water you plan to store will not take even half of your smaller drum
- eldenring 2y ago
  No their claim is that there are dimishing returns for a fixed compute budget (in training) to scaling up data past that threshold vs. scaling up params. This doesn't take inference into account either, obviously.
deleted 2y ago
[deleted]

moffkalast 2y ago

Their rule of thumb would imply that a 70B model is saturated with 1.7T tokens, that's inconsistent with reality.

famouswaffles 2y ago
The Chinchilla laws were compute optimal scaling laws. They're not supposed to tell you what parameter-token combination will saturate a model.
- moffkalast 2y ago
  Compute optimal for what, training? There's nothing optimal in blowing up model size beyond the absolute minimal needed or you'll spent the equivalent of a country in electricity trying to scale inference later.
  - rfw300 2y ago
    Yes, compute-optimal for training only. The purpose of the paper wasn’t to determine the most economically practical model one could build, the most “intelligent” model one could build given some amount of compute.
    - ijk 2y ago
      Quite. The big question at the time was "how much data do we need to train GPT-3 equivalent models". Open models had failed to live up to GPT performance, even ones with a massive number of parameters. So getting results that suggested a reason why other models were massively undertrained was important. Meanwhile, people noticed that for deployed models, inference cost often outweighs the initial training costs. It's sometimes better to train a smaller, faster model longer on more data, because it has lower overall cost (including environmental impact) if you're expecting to run the model a few million or billion times (e.g., [1]). So training past the Chinchilla optimum point became a lot more common, particularly after Llama. [1] https://arxiv.org/abs/2401.00448 https://arxiv.org/abs/2401.00448
  - FeepingCreature 2y ago
    Blow up model size, get lots of space and parameters to do the double-descent grok thing in, then distill it way way down?
  - famouswaffles 2y ago
    Training yes. Doubling your parameter count past that ratio will yield a better model than doubling your data and is much easier and cheaper to do.
    - naasking 2y ago
      That suggests that it's likely memorizing more special cases rather than distilling general principles. They generalize to some degree but clearly there's room for improvement.
      - famouswaffles 2y ago
        It doesn't really suggest anything. Neither model will even close to saturation and all else equal, bigger models perform better in every way, including generalization.
        naasking 2y ago
        But why do bigger models perform better? Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing.
        famouswaffles 2y ago
        >But why do bigger models perform better? No one really knows the answer to this question. >Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing. What I'm trying to say is that both models in either scenario are very over-parameterized and under-trained. You say the answer is extra space ? The smaller model has not used anywhere near the space it has. They both have extra space. It's like arguing a bigger drum is better because of extra space when all the water you plan to store will not take even half of your smaller drum
eldenring 2y ago
No their claim is that there are dimishing returns for a fixed compute budget (in training) to scaling up data past that threshold vs. scaling up params. This doesn't take inference into account either, obviously.

deleted 2y ago

[deleted]