3 ms·

This amount of diversity in transformers is very impressive, but what’s more impressive is that for models like GPT, scaling the models seems much more effectiv

This amount of diversity in transformers is very impressive, but what’s more impressive is that for models like GPT, scaling the models seems much more effective than engineering the models

alexmolas 4y ago
I don't remember the paper, I think it's on the vision transformers paper, that they say something like "scaling the model and having more data completely beats inductive bias". It's impressive how we went from feature engineering in classical ML, to inductive bias in early deep learning, to just have more data in modern deep learning.
- 0xBABAD00C 4y ago
  > scaling the model and having more data completely beats inductive bias The analogy in my mind is this: "burning natural oil/gas completely beats figuring out cleaner & more sustainable energy sources" My point is that "more data" here simply represents the mental effort that has already been exerted in pre-AI/DL era, which we're now capitalizing on while we can. Similar to how fossil fuels represent the energy storage efforts by earlier lifeforms that we're now capitalizing on, again while we can. It's a system way out of equilibrium, progressing while it can on borrowed resources from the prior generations. In the long run, the AI agents will be less wasteful as they reach the limits of what data or energy is available on the margins to compete within themselves and to reach their goals. It's just we haven't reached that limit yet, and the competition at this stage is on processing more data and scaling the models at any cost.
  - chaxor 4y ago
    Perhaps another analogy is if you train something by repeatedly telling it many different stories about the same thing day in and day out, compared to mentioning something just once in passing, perhaps the system will know what it's been exposed to more. Replaying that event it was exposed to in passing in order to check it for parsimony requires more mental effort and seems like something that requires explicitly setting aside the work to do so.
  - sebzim4500 4y ago
    >My point is that "more data" here simply represents the mental effort that has already been exerted in pre-AI/DL era, which we're now capitalizing on while we can Not really. It's not simply that modern architectures are not adding additional inductive biases, they are actively throwing away the inductive bias that used to be used by everyone. For example, it was taken for granted that you should use CNNs to give you translation invariance, but apparently now visual transformers can match that performance with the same amount of compute.
    - 0xBABAD00C 4y ago
      I don't think what you're saying contradicts what I'm saying. My baseline / reference point wasn't CNNs.
    - jerpint 4y ago
      There are bit caveats here: Vision transformers outperform CNNs in HUGE data regimes. On small datasets, CNNs still shine. Also, if you take a CNN with modern tricks, they can be on par with vision transformers, e.g convnext Transformers really dominate when you scale the amount of data to infinity
      - alexmolas 4y ago
        Yes, that's what I was saying: scale > inductive bias. Apparently, scaling is all you need ;)
ad404b8a372f2b9 4y ago
I see no evidence of that, transformers seem to follow the same trend as other architectures with improved models coming out every month that demonstrate similar performance with orders of magnitude less parameters.
- nerdponx 4y ago
  I think by "effective" they meant better overall performance, not necessarily better performance per parameter or hour of training.

This amount of diversity in transformers is very impressive, but what’s more impressive is that for models like GPT, scaling the models seems much more effective than engineering the models

alexmolas 4y ago

I don't remember the paper, I think it's on the vision transformers paper, that they say something like "scaling the model and having more data completely beats inductive bias". It's impressive how we went from feature engineering in classical ML, to inductive bias in early deep learning, to just have more data in modern deep learning.

0xBABAD00C 4y ago
> scaling the model and having more data completely beats inductive bias The analogy in my mind is this: "burning natural oil/gas completely beats figuring out cleaner & more sustainable energy sources" My point is that "more data" here simply represents the mental effort that has already been exerted in pre-AI/DL era, which we're now capitalizing on while we can. Similar to how fossil fuels represent the energy storage efforts by earlier lifeforms that we're now capitalizing on, again while we can. It's a system way out of equilibrium, progressing while it can on borrowed resources from the prior generations. In the long run, the AI agents will be less wasteful as they reach the limits of what data or energy is available on the margins to compete within themselves and to reach their goals. It's just we haven't reached that limit yet, and the competition at this stage is on processing more data and scaling the models at any cost.
- chaxor 4y ago
  Perhaps another analogy is if you train something by repeatedly telling it many different stories about the same thing day in and day out, compared to mentioning something just once in passing, perhaps the system will know what it's been exposed to more. Replaying that event it was exposed to in passing in order to check it for parsimony requires more mental effort and seems like something that requires explicitly setting aside the work to do so.
- sebzim4500 4y ago
  >My point is that "more data" here simply represents the mental effort that has already been exerted in pre-AI/DL era, which we're now capitalizing on while we can Not really. It's not simply that modern architectures are not adding additional inductive biases, they are actively throwing away the inductive bias that used to be used by everyone. For example, it was taken for granted that you should use CNNs to give you translation invariance, but apparently now visual transformers can match that performance with the same amount of compute.
  - 0xBABAD00C 4y ago
    I don't think what you're saying contradicts what I'm saying. My baseline / reference point wasn't CNNs.
  - jerpint 4y ago
    There are bit caveats here: Vision transformers outperform CNNs in HUGE data regimes. On small datasets, CNNs still shine. Also, if you take a CNN with modern tricks, they can be on par with vision transformers, e.g convnext Transformers really dominate when you scale the amount of data to infinity
    - alexmolas 4y ago
      Yes, that's what I was saying: scale > inductive bias. Apparently, scaling is all you need ;)

ad404b8a372f2b9 4y ago

I see no evidence of that, transformers seem to follow the same trend as other architectures with improved models coming out every month that demonstrate similar performance with orders of magnitude less parameters.

nerdponx 4y ago
I think by "effective" they meant better overall performance, not necessarily better performance per parameter or hour of training.