4 ms·

Reading this, all the changes in modern models like gpt-3 seem like pretty incremental improvements - better structured training data (and a lot more of it), us

Reading this, all the changes in modern models like gpt-3 seem like pretty incremental improvements - better structured training data (and a lot more of it), using tokens instead of characters, etc.

loveparade 3y ago
They are. Nobody in the AI research community expected that scaling up models several orders of magnitude would make them that good. Even the original selling point of the Transformer paper was "can be much bigger and faster than RNNs, even if not as good" - Of course everyone was wrong and it's all about the data and scale. Btw, tokens are nothing new in modern models. They've been used since 2015/2016 [0] [0] https://arxiv.org/abs/1508.07909 https://arxiv.org/abs/1508.07909
- williamtrask 3y ago
  Which goes to show you one of the best strategies in ML isn’t necessarily thinking about learning theory, it’s thinking about what our fastest computers are most capable of doing. But this does run out of steam (or rather…data) eventually. The largest LLM was ibm’s tangora model back in the 1980s. 8 trillion params. We’ll go through the cycle again with a new paradigm.
  - IIAOPSW 3y ago
    "It just goes to show the best strategy in assembling wood together isn't learning to use a screw driver, its about pounding it in with the hardest and most capable hammer."
    - simonh 3y ago
      Yep. Sometimes so. https://www.thesprucecrafts.com/wood-joinery-methods-use-no-fasteners-3536634 https://www.thesprucecrafts.com/wood-joinery-methods-use-no-...
      - IIAOPSW 3y ago
        I'll acknowledge Large Hammer Models have done surprisingly well, but true General Artificial Fastening is obviously not just Hammer models scaled up. There's things a screw driver can do which a hammer is hopeless on. Such as taking apart your eye glasses (and more importantly, putting them back together). This isn't something that is going to be solved once the tradesmen have a bit more training. Its fundamental to the way hammers work. We can only do so much by hammering in all our screws.
  - yarg 3y ago
    There was a similar argument for not improving the efficiency of software in general - why bother to improve the design if upgrading the underlying hardware can double performance without changing the software? It's not entirely invalid, but it does encourage a rather blasé attitude towards actually understanding the problem that you're dealing with.
    - balder1991 3y ago
      And at some point the cost of hardware starts to add up.
  - tornato7 3y ago
    8 trillion params in 1980? I don't think so bud. There wasn't that much memory in the entire world at the time.
  - deleted 3y ago
    [deleted]
  - creatonez 3y ago
    > The largest LLM was ibm’s tangora model back in the 1980s. 8 trillion params. As far as I can tell, this comes from a typo in a 1992 paper? https://aclanthology.org/J92-4003.pdf https://aclanthology.org/J92-4003.pdf Does it mean the space of possible inputs is 8 trillion?
    - gwern 3y ago
      It's an n-gram model so it can easily have as many parameters as you want to store. Skimming it, I think the '8 trillion parameters' here means '8 trillion trainable n-gram parameters but it's very sparse and most of them are simply defined by omission to be a small constant like epsilon due to not being represented in the training data at all and so suffering from the 0-count problem'. (Old problem - even Turing and Good spent a lot of time discussing how to handle 0s in contingency tables because that happens a lot in cryptography!)
    - IshanMi 3y ago
      I think the 8 trillion parameters is accurate- Tangora is an N-gram model with a vocab size of 20,000 words and N = 3. Parameters for an N-gram model = V^(N-1) * (V-1) Plugging in V=20,000 words and N = 3 for Tangora, you'd get 7.9996E12. Most of the parameters are likely zero or close to it because many 3-grams are possible but not likely to occur. (However the aggregate probability of all 3-grams is substantial and thus they have to be included.)
creatonez 3y ago
I'm not sure where it was originally used, but the attention mechanism in GPT was pretty important for making it all work.
- williamtrask 3y ago
  Paper you’re looking for is called “Attention is all you need”
gwern 3y ago
That's because the big change was moving from LSTM RNNs to Transformers. If you had tried to train a scaled-up RNN on GPT-3-level compute and data, it would have been a much better RNN, yes... and terrible compared to GPT-3, because the RNN scaling law is much worse and they are unable to make good use of context. See Kaplan et al 2020, key graph: https://gwern.net/doc/ai/nn/transformer/gpt/2020-kaplan-figure7-rnnsvstransformers.png https://gwern.net/doc/ai/nn/transformer/gpt/2020-kaplan-figu... (I'm not going to try to calculate it, but by 'terrible' I mean that the bent scaling law would give you, like, GPT-2 performance for GPT-3-level investment. Suffice it to say that the DL conversations would be a trifle different today if that had been what happened.) All the other stuff has been largely minor polishes. For example, using tokens instead of characters? Nah. ByT5 works fine, as do very large context window approaches like Jukebox. Large BPE vocabs may be somewhat more efficient but the value is nothing like RNN->Transformer. A lot more training data? Doesn't matter if your model scaling law is bent because the model can't make use of more data meaningfully - it's not like it's overfitting to smaller amounts of data either, it's just underfitting everything. Structure? Irrelevant, the point of web scrapes is bulk and diversity. etc
- rolisz 3y ago
  Well, RNNs are making a comeback: https://lmsys.org/blog/2023-05-10-leaderboard/ https://lmsys.org/blog/2023-05-10-leaderboard/ RWKV is an RNN based model and it's competitive (with the open source models, not with GPT4)
  - gwern 3y ago
    RWKV is interesting but hasn't shown Transformer-beating improvement yet, and its key is that it's training in a quasi-Transformer manner to begin with, which is why it gets the better scaling law that makes it feasible, and so demonstrates my point. (RWKV is more like a Transformer which is easily converted into a RNN for sampling, which is an equivalence we've known for a long time.)
    - rolisz 3y ago
      It's beaten by only two weights available models, it beats LLama 13B and Dolly. What do you mean it's training in a quasi-Transformer manner? Do you mean the language pretraining? That's been done before Transformers. RWKV doesn't have the quadratic attention of Transformers, so I'd say it's pretty different.

Reading this, all the changes in modern models like gpt-3 seem like pretty incremental improvements - better structured training data (and a lot more of it), using tokens instead of characters, etc.

loveparade 3y ago

They are. Nobody in the AI research community expected that scaling up models several orders of magnitude would make them that good. Even the original selling point of the Transformer paper was "can be much bigger and faster than RNNs, even if not as good" - Of course everyone was wrong and it's all about the data and scale. Btw, tokens are nothing new in modern models. They've been used since 2015/2016 [0] [0] https://arxiv.org/abs/1508.07909 https://arxiv.org/abs/1508.07909

williamtrask 3y ago
Which goes to show you one of the best strategies in ML isn’t necessarily thinking about learning theory, it’s thinking about what our fastest computers are most capable of doing. But this does run out of steam (or rather…data) eventually. The largest LLM was ibm’s tangora model back in the 1980s. 8 trillion params. We’ll go through the cycle again with a new paradigm.
- IIAOPSW 3y ago
  "It just goes to show the best strategy in assembling wood together isn't learning to use a screw driver, its about pounding it in with the hardest and most capable hammer."
  - simonh 3y ago
    Yep. Sometimes so. https://www.thesprucecrafts.com/wood-joinery-methods-use-no-fasteners-3536634 https://www.thesprucecrafts.com/wood-joinery-methods-use-no-...
    - IIAOPSW 3y ago
      I'll acknowledge Large Hammer Models have done surprisingly well, but true General Artificial Fastening is obviously not just Hammer models scaled up. There's things a screw driver can do which a hammer is hopeless on. Such as taking apart your eye glasses (and more importantly, putting them back together). This isn't something that is going to be solved once the tradesmen have a bit more training. Its fundamental to the way hammers work. We can only do so much by hammering in all our screws.
- yarg 3y ago
  There was a similar argument for not improving the efficiency of software in general - why bother to improve the design if upgrading the underlying hardware can double performance without changing the software? It's not entirely invalid, but it does encourage a rather blasé attitude towards actually understanding the problem that you're dealing with.
  - balder1991 3y ago
    And at some point the cost of hardware starts to add up.
- tornato7 3y ago
  8 trillion params in 1980? I don't think so bud. There wasn't that much memory in the entire world at the time.
- deleted 3y ago
  [deleted]
- creatonez 3y ago
  > The largest LLM was ibm’s tangora model back in the 1980s. 8 trillion params. As far as I can tell, this comes from a typo in a 1992 paper? https://aclanthology.org/J92-4003.pdf https://aclanthology.org/J92-4003.pdf Does it mean the space of possible inputs is 8 trillion?
  - gwern 3y ago
    It's an n-gram model so it can easily have as many parameters as you want to store. Skimming it, I think the '8 trillion parameters' here means '8 trillion trainable n-gram parameters but it's very sparse and most of them are simply defined by omission to be a small constant like epsilon due to not being represented in the training data at all and so suffering from the 0-count problem'. (Old problem - even Turing and Good spent a lot of time discussing how to handle 0s in contingency tables because that happens a lot in cryptography!)
  - IshanMi 3y ago
    I think the 8 trillion parameters is accurate- Tangora is an N-gram model with a vocab size of 20,000 words and N = 3. Parameters for an N-gram model = V^(N-1) * (V-1) Plugging in V=20,000 words and N = 3 for Tangora, you'd get 7.9996E12. Most of the parameters are likely zero or close to it because many 3-grams are possible but not likely to occur. (However the aggregate probability of all 3-grams is substantial and thus they have to be included.)

creatonez 3y ago

I'm not sure where it was originally used, but the attention mechanism in GPT was pretty important for making it all work.

williamtrask 3y ago
Paper you’re looking for is called “Attention is all you need”

gwern 3y ago

That's because the big change was moving from LSTM RNNs to Transformers. If you had tried to train a scaled-up RNN on GPT-3-level compute and data, it would have been a much better RNN, yes... and terrible compared to GPT-3, because the RNN scaling law is much worse and they are unable to make good use of context. See Kaplan et al 2020, key graph: https://gwern.net/doc/ai/nn/transformer/gpt/2020-kaplan-figure7-rnnsvstransformers.png https://gwern.net/doc/ai/nn/transformer/gpt/2020-kaplan-figu... (I'm not going to try to calculate it, but by 'terrible' I mean that the bent scaling law would give you, like, GPT-2 performance for GPT-3-level investment. Suffice it to say that the DL conversations would be a trifle different today if that had been what happened.) All the other stuff has been largely minor polishes. For example, using tokens instead of characters? Nah. ByT5 works fine, as do very large context window approaches like Jukebox. Large BPE vocabs may be somewhat more efficient but the value is nothing like RNN->Transformer. A lot more training data? Doesn't matter if your model scaling law is bent because the model can't make use of more data meaningfully - it's not like it's overfitting to smaller amounts of data either, it's just underfitting everything. Structure? Irrelevant, the point of web scrapes is bulk and diversity. etc

rolisz 3y ago
Well, RNNs are making a comeback: https://lmsys.org/blog/2023-05-10-leaderboard/ https://lmsys.org/blog/2023-05-10-leaderboard/ RWKV is an RNN based model and it's competitive (with the open source models, not with GPT4)
- gwern 3y ago
  RWKV is interesting but hasn't shown Transformer-beating improvement yet, and its key is that it's training in a quasi-Transformer manner to begin with, which is why it gets the better scaling law that makes it feasible, and so demonstrates my point. (RWKV is more like a Transformer which is easily converted into a RNN for sampling, which is an equivalence we've known for a long time.)
  - rolisz 3y ago
    It's beaten by only two weights available models, it beats LLama 13B and Dolly. What do you mean it's training in a quasi-Transformer manner? Do you mean the language pretraining? That's been done before Transformers. RWKV doesn't have the quadratic attention of Transformers, so I'd say it's pretty different.