4 ms·

I'm not sure folks who're putting out strong takes based on this have read this paper. This paper uses GPT-2 transformer scale, on sinusoidal data: >We traine

I'm not sure folks who're putting out strong takes based on this have read this paper. This paper uses GPT-2 transformer scale, on sinusoidal data: >We trained a decoder-only Transformer [7] model of GPT-2 scale implemented in the Jax based machine learning framework, Pax4 with 12 layers, 8 attention heads, and a 256-dimensional embedding space (9.5M parameters) as our base configuration [4]. > Building on previous work, we investigate this question in a controlled setting, where we study transformer models trained on sequences of (x,f(x)) pairs rather than natural language. Nowhere near definitive or conclusive. Not sure why this is news outside of the Twitter-techno-pseudo-academic-influencer bubble.

andy99 3y ago
It would be news is somebody showed transformers could generalize beyond the training data. Deep learning models generally cannot, so it's not a surprise this holds for transformers.
- GaggiX 3y ago
  It depends on what does "generalize beyond the training data" means. If I invent a new programming language and I teach (in-context) the language to the model and it's able to use it to solve many tasks, is it generalizing beyond the training data?
  - andy99 3y ago
    No. The way I'd look at it is that generalization or specifically extrapolation would mean that different features are needed to make a prediction (here, the next token) than what is seen in the training data. Something like a made up language could still result in the same patterns being relevant. That's why out-of-distribution research often uses mathematical extrapolation as a task.
    - famouswaffles 3y ago
      "Supercharged Interpolation" is not something that actually exists. Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 What you're asking for is not "generalization" but magic and humans would also fail.
      - deleted 3y ago
        [deleted]
    - spywaregorilla 3y ago
      Can you provide a real world example? Because this sounds like nonsense. As in, not a weakness of any architecture but just the very concept of pattern matching. What you might be asking for is a system that simply continually learns.
    - vidarh 3y ago
      Can humans do this? Because I'm not convinced humans can do this. Or that it reasonably means anything.
- Swizec 3y ago
  > somebody showed transformers could generalize I read an interesting paper recently that had a great take on this: If you add enough data, nothing is outside training data. Thus solving the generalization problem. Wasn’t the main point of that paper, but it made me go ”Huh yeah … I guess … technically correct?”. It raises an interesting thought that yes if you just train your neural network on everything, then nothing falls outside its domain. Problem solved … now if only compute was cheap.
  - andy99 3y ago
    Compute and data collection. This is the long tail problem.
    - sharemywin 3y ago
      doesn't the long tail just need to be a little outside the range of human capability for it to be AGI?
      - andy99 3y ago
        Not sure I understand but people don't need the long tail because we don't write rules and then blindly act on them when we encounter new things. We can reason about stuff we haven't seen before.
- cosmojg 3y ago
  Has it even been shown that the average human can generalize beyond their training data? Isn't this the central thrust of the controversy around IQ tests? For example, some argue that access to relevant training data is a greater determinant of performance on IQ tests than genetics[1]. [1] https://www.youtube.com/watch?v=FkKPsLxgpuY https://www.youtube.com/watch?v=FkKPsLxgpuY
  - sudosysgen 3y ago
    Just because we can be genuinely creative in some way doesn't mean that having access to data doesn't make it easier.
    - cosmojg 3y ago
      Indeed! And even then, can we be genuinely creative? As far as I can tell, everything is derivative.
      - czzr 3y ago
        Considering the human race bootstrapped itself from nothing, what’s your mental model here? Where did everything come from?
        cosmojg 3y ago
        Remembering the outcomes of accidents and remixing those memories to produce derivative works.
        kridsdale3 3y ago
        Nature.
        throwaway092323 3y ago
        Humans and AIs both evolve as the result of some iterations dying. In both cases, we tacitly erase the ones who don't make it (by framing the discussion around the successful, alive ones). The difference is that humans have had a broader training set.
- YetAnotherNick 3y ago
  Define "generalize beyond training data" please. Because in my opinion asking a model to produce avacado chair is generalizing beyond training data.
- huijzer 3y ago
  OpenAI showed it in 2017 with the sentiment neuron (https://openai.com/research/unsupervised-sentiment-neuron https://openai.com/research/unsupervised-sentiment-neuron). Basically, the model learned to classify the sentiment of a text which I would agree is a general principle, so the model learned a generalized representation based on the data. Having said that, the real question is what percentage of the learned representations do generalize. For a perfect model, it would learn only representations that generalize and none that overfit. But, that's unreasonable to expect for a machine *and* even for a human. Maybe we just don't know. We are staring at a black box and doing some statistical tests, but actually don't know whether the current AI architecture is capable enough to get to some kind of human intelligence equivalent.
quotemstr 3y ago
> Not sure why this is news outside of the Twitter-techno-pseudo-academic-influencer bubble. The paper is making the rounds despite being a weak result because it confirms what people want, for non-technical reasons, to be true. You see this kind of thing all the time in other fields: for decades, the media has elevated p-hacked psychology studies on three undergrads into the canon of pop psychology because these studies provide a fig leaf of objective backing for pre-determined conclusions
deleted 3y ago
[deleted]
gwern 3y ago
> I'm not sure folks who're putting out strong takes based on this have read this paper. They haven't read the other papers either. It's really striking to me to watch people retweet this and it get written up in pseudo-media like Business Insider when other meta-learning papers on the distributional hypothesis of inducing meta-learning & generalization, which are at least as relevant, can't even make a peep on specialized research subreddits - like, "Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression", Raventós et al 2023 https://arxiv.org/abs/2306.15063 https://arxiv.org/abs/2306.15063 (or https://arxiv.org/abs/2310.08391 https://arxiv.org/abs/2310.08391 ) both explains & obsoletes OP, and it was published months before! OP is a highly limited result which doesn't actually show anything that you wouldn't expect on ordinary Bayesian meta-reinforcement-learning grounds, but there's so much appetite for someone claiming that this time, for real, DL will 'hit the wall' that any random paper appears to be definitive to critics.

andy99 3y ago

It would be news is somebody showed transformers could generalize beyond the training data. Deep learning models generally cannot, so it's not a surprise this holds for transformers.

GaggiX 3y ago
It depends on what does "generalize beyond the training data" means. If I invent a new programming language and I teach (in-context) the language to the model and it's able to use it to solve many tasks, is it generalizing beyond the training data?
- andy99 3y ago
  No. The way I'd look at it is that generalization or specifically extrapolation would mean that different features are needed to make a prediction (here, the next token) than what is seen in the training data. Something like a made up language could still result in the same patterns being relevant. That's why out-of-distribution research often uses mathematical extrapolation as a task.
  - famouswaffles 3y ago
    "Supercharged Interpolation" is not something that actually exists. Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 What you're asking for is not "generalization" but magic and humans would also fail.
    - deleted 3y ago
      [deleted]
  - spywaregorilla 3y ago
    Can you provide a real world example? Because this sounds like nonsense. As in, not a weakness of any architecture but just the very concept of pattern matching. What you might be asking for is a system that simply continually learns.
  - vidarh 3y ago
    Can humans do this? Because I'm not convinced humans can do this. Or that it reasonably means anything.
Swizec 3y ago
> somebody showed transformers could generalize I read an interesting paper recently that had a great take on this: If you add enough data, nothing is outside training data. Thus solving the generalization problem. Wasn’t the main point of that paper, but it made me go ”Huh yeah … I guess … technically correct?”. It raises an interesting thought that yes if you just train your neural network on everything, then nothing falls outside its domain. Problem solved … now if only compute was cheap.
- andy99 3y ago
  Compute and data collection. This is the long tail problem.
  - sharemywin 3y ago
    doesn't the long tail just need to be a little outside the range of human capability for it to be AGI?
    - andy99 3y ago
      Not sure I understand but people don't need the long tail because we don't write rules and then blindly act on them when we encounter new things. We can reason about stuff we haven't seen before.
cosmojg 3y ago
Has it even been shown that the average human can generalize beyond their training data? Isn't this the central thrust of the controversy around IQ tests? For example, some argue that access to relevant training data is a greater determinant of performance on IQ tests than genetics[1]. [1] https://www.youtube.com/watch?v=FkKPsLxgpuY https://www.youtube.com/watch?v=FkKPsLxgpuY
- sudosysgen 3y ago
  Just because we can be genuinely creative in some way doesn't mean that having access to data doesn't make it easier.
  - cosmojg 3y ago
    Indeed! And even then, can we be genuinely creative? As far as I can tell, everything is derivative.
    - czzr 3y ago
      Considering the human race bootstrapped itself from nothing, what’s your mental model here? Where did everything come from?
      - cosmojg 3y ago
        Remembering the outcomes of accidents and remixing those memories to produce derivative works.
      - kridsdale3 3y ago
        Nature.
      - throwaway092323 3y ago
        Humans and AIs both evolve as the result of some iterations dying. In both cases, we tacitly erase the ones who don't make it (by framing the discussion around the successful, alive ones). The difference is that humans have had a broader training set.
YetAnotherNick 3y ago
Define "generalize beyond training data" please. Because in my opinion asking a model to produce avacado chair is generalizing beyond training data.
huijzer 3y ago
OpenAI showed it in 2017 with the sentiment neuron (https://openai.com/research/unsupervised-sentiment-neuron https://openai.com/research/unsupervised-sentiment-neuron). Basically, the model learned to classify the sentiment of a text which I would agree is a general principle, so the model learned a generalized representation based on the data. Having said that, the real question is what percentage of the learned representations do generalize. For a perfect model, it would learn only representations that generalize and none that overfit. But, that's unreasonable to expect for a machine *and* even for a human. Maybe we just don't know. We are staring at a black box and doing some statistical tests, but actually don't know whether the current AI architecture is capable enough to get to some kind of human intelligence equivalent.

quotemstr 3y ago

> Not sure why this is news outside of the Twitter-techno-pseudo-academic-influencer bubble. The paper is making the rounds despite being a weak result because it confirms what people want, for non-technical reasons, to be true. You see this kind of thing all the time in other fields: for decades, the media has elevated p-hacked psychology studies on three undergrads into the canon of pop psychology because these studies provide a fig leaf of objective backing for pre-determined conclusions

deleted 3y ago

[deleted]

gwern 3y ago

> I'm not sure folks who're putting out strong takes based on this have read this paper. They haven't read the other papers either. It's really striking to me to watch people retweet this and it get written up in pseudo-media like Business Insider when other meta-learning papers on the distributional hypothesis of inducing meta-learning & generalization, which are at least as relevant, can't even make a peep on specialized research subreddits - like, "Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression", Raventós et al 2023 https://arxiv.org/abs/2306.15063 https://arxiv.org/abs/2306.15063 (or https://arxiv.org/abs/2310.08391 https://arxiv.org/abs/2310.08391 ) both explains & obsoletes OP, and it was published months before! OP is a highly limited result which doesn't actually show anything that you wouldn't expect on ordinary Bayesian meta-reinforcement-learning grounds, but there's so much appetite for someone claiming that this time, for real, DL will 'hit the wall' that any random paper appears to be definitive to critics.