3 ms·

One major caveat here: the Chinchilla scaling law indicates that most of this compute was wasted. You could've gotten PaLM's performance at a fraction of the co

by gwern 5y ago

One major caveat here: the Chinchilla scaling law indicates that most of this compute was wasted. You could've gotten PaLM's performance at a fraction of the cost - Chinchilla approaches PaLM performance but at a fifth the cost.

_hark 5y ago
DeepMind to OpenAI and everyone else[1]: > your hyperparameters are bad and you should feel bad It's amazing to me that such a big goof was missed by so many for so long. All these multimillion dollar language models and people just took the scaling laws at face value. [1]: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556
- gwern 5y ago
  Isn't it amazing? One of those reminders that we don't understand DL as much as we like to pretend we do. Cyclic learning rates work well elsewhere, but I can't think of any other case where switching made such a difference. I was completely shocked to read Chinchilla, and I'm still a little baffled - a cosine schedule? Really? That's it? You guys didn't make any other changes?
timpetri 5y ago
Question related to the Chinchilla paper[0], which says that optimal amount of training data for ~500B, 1T, and 10T param models are 11T, 21.2T, 216.2T tokens, respectively. The PaLM paper[1] says it made use of 700B tokens. How many tokens of training data have humans produced across the entire internet, all our written works, etc? Is there such a thing as a 216 trillion token set? [0] https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 [1] https://arxiv.org/abs/2204.02311 https://arxiv.org/abs/2204.02311
- gwern 5y ago
  Humans produce an astonishing amount of text if you consider all the source code, research data, social media websites, emails etc and project out a decade or two; there is also multimodal and RL to consider as a source of 'tokens' like visual tokens, which have ~infinite data. Text is great, but there is no reason to train only text. It's just a good starting point. But the real question you should be asking is, where would you get the compute to train a model that needs 216t tokens?
nootropicat 5y ago
It was only 'wasted' if the intention was to make a commercial model, not a research model. Google could afford a 100x larger model (than PaLM), a 1000x one would probably require a consortium including other corporations and possibly governments. Chinchilla used 1.4T tokens - is there even 100x more text data available, in any practical sense - as in potentially available for training? As long as the model size scaling improves performance alone, it makes sense to scale. Only once performance saturates, given same data, it's time to switch attention to training on as much data as possible - and hopefully such model turns out to be superhuman in writing code - improving itself on the algorithmic side, starting the singularity. As an additional point, an AGI in this approach may well need to be trained on images and videos (both containing text in addition to other information) - which requires way more compute power. This would make enormous 'wasteful' text models a way to reach a point where enormous future models and training architectures can be reused for visual and sound data with small changes.

by gwern 5y ago

_hark 5y ago

DeepMind to OpenAI and everyone else[1]: > your hyperparameters are bad and you should feel bad It's amazing to me that such a big goof was missed by so many for so long. All these multimillion dollar language models and people just took the scaling laws at face value. [1]: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556

gwern 5y ago
Isn't it amazing? One of those reminders that we don't understand DL as much as we like to pretend we do. Cyclic learning rates work well elsewhere, but I can't think of any other case where switching made such a difference. I was completely shocked to read Chinchilla, and I'm still a little baffled - a cosine schedule? Really? That's it? You guys didn't make any other changes?

timpetri 5y ago

Question related to the Chinchilla paper[0], which says that optimal amount of training data for ~500B, 1T, and 10T param models are 11T, 21.2T, 216.2T tokens, respectively. The PaLM paper[1] says it made use of 700B tokens. How many tokens of training data have humans produced across the entire internet, all our written works, etc? Is there such a thing as a 216 trillion token set? [0] https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 [1] https://arxiv.org/abs/2204.02311 https://arxiv.org/abs/2204.02311

gwern 5y ago
Humans produce an astonishing amount of text if you consider all the source code, research data, social media websites, emails etc and project out a decade or two; there is also multimodal and RL to consider as a source of 'tokens' like visual tokens, which have ~infinite data. Text is great, but there is no reason to train only text. It's just a good starting point. But the real question you should be asking is, where would you get the compute to train a model that needs 216t tokens?

nootropicat 5y ago

It was only 'wasted' if the intention was to make a commercial model, not a research model. Google could afford a 100x larger model (than PaLM), a 1000x one would probably require a consortium including other corporations and possibly governments. Chinchilla used 1.4T tokens - is there even 100x more text data available, in any practical sense - as in potentially available for training? As long as the model size scaling improves performance alone, it makes sense to scale. Only once performance saturates, given same data, it's time to switch attention to training on as much data as possible - and hopefully such model turns out to be superhuman in writing code - improving itself on the algorithmic side, starting the singularity. As an additional point, an AGI in this approach may well need to be trained on images and videos (both containing text in addition to other information) - which requires way more compute power. This would make enormous 'wasteful' text models a way to reach a point where enormous future models and training architectures can be reused for visual and sound data with small changes.