4 ms·

Yes even the original Transformers model had only millions of parameters and nonetheless showed "impressive capabilities" because it also had Self-Attention. I

Yes even the original Transformers model had only millions of parameters and nonetheless showed "impressive capabilities" because it also had Self-Attention. If you know of any models that have had success (even at the GPT-2 level) without Self-Attention, I'd be interested to know what they are, because I don't know of any.

littlestymaar 2y ago
RWKV. There aren't many multi-billion-parameters non-transformer models because of path dependence, but that doesn't mean that only transformers can achieve this kind of results.
- quantadev 2y ago
  My statements (which you disagreed with, without exception) haven't been about Transformers v.s. non-Transformers. Everything above has been about the importance of the Self-Attention part of it. We could remove Self-Attention from Transformers and still have a functional (but dumb) NN, and that was my point. Your position was that the Self-Attention is a less important part (because UAT, yadda yadda), and my position was that it's the key ingredient. Every statement above that I made, that you called wrong, was correct. lol.
  - littlestymaar 2y ago
    You are moving the goalpost. The discussion has always been about transformers vs non transformers. You claimed that self attention was needed to achieve the level of intelligence that we've seen with GPT 3.5: > without those attention heads even the scaling up to current parameter sizes we have to day would not have ended up with the level of emergent intelligence that shocked the world with GPT 3.5. (Verbatim quote from you https://news.ycombinator.com/item?id=41986010 https://news.ycombinator.com/item?id=41986010) This is the claim I've been disputing, by responding that the key feature of the intelligence of tranformer models come from their scalability. And now that we have alternative that scale equally well (SSM and RWKV) unsurprisingly we see them achieve the same level of reasoning abilities. > Every statement above that I made, that you called wrong, was correct. lol. Well, except the one quoted above at least…
    - quantadev 2y ago
      In the quote you're calling wrong (41986010), you're interpreting "scaling up" as "scaling up, including changing architecture". Scaling up transformers just means scaling up transformers, and keeping everything else the same. In other words you're interpreting "parameter size" as "parameter size, independent of architecture", and I meant parameter size of a Transformer (in the context of with v.s. without Self-Attention).
      - littlestymaar 2y ago
        Pathetic.
        quantadev 2y ago
        Straw-manning failed, so now you insult.
        littlestymaar 2y ago
        There's no staw-man, and you are now at the point of trying to re-invent the definition of words in order to somehow “win the argument ” without even respecting your own previous position. This behavior is legit pathetic, it's not an insult it's a fact. Respect yourself.
        quantadev 2y ago
        I stand by every word: 1) Self-Attention is more important than scale, and 2) to test that claim, simply remove SA from a transformer and see if it destroys the "intelligence" or not. There's nothing confusing about that, but thanks for your concerns and your polite words.
        littlestymaar 2y ago
        No that wasn't your argument and this new one is off course a much waker one that you fell back onto to be “technically right”. That attention heads are mandatory for transformers is a tautology (without it a transformer is just an MLP…) so of course this statement is going to be correct, by definition. But when you move the goal post to land on a tautology then you've surrendered your abilities to argue anything and you are just ridiculing yourself. Take this question of your for instance: > If you know of any models that have had success (even at the GPT-2 level) without Self-Attention, I'd be interested to know what they are, because I don't know of any. Which is a legit, non-ridiculous, one. If you replace it with your later much weaker argument: > > If you know of any MLP that have had success (even at the GPT-2 level), I'd be interested to know what they are, because I don't know of any. Then it becomes a dumb question given that MLP have no way of encoding context and can't process sequences of words in the first place. So when you argue that it was your argument all along, it's particularly embarrassing because you're just arguing that your previous arguments were equally dumb even when they weren't. That's why I said you're disrespecting your earlier argumentation by retreating to your later tautology.

littlestymaar 2y ago

RWKV. There aren't many multi-billion-parameters non-transformer models because of path dependence, but that doesn't mean that only transformers can achieve this kind of results.

quantadev 2y ago
My statements (which you disagreed with, without exception) haven't been about Transformers v.s. non-Transformers. Everything above has been about the importance of the Self-Attention part of it. We could remove Self-Attention from Transformers and still have a functional (but dumb) NN, and that was my point. Your position was that the Self-Attention is a less important part (because UAT, yadda yadda), and my position was that it's the key ingredient. Every statement above that I made, that you called wrong, was correct. lol.
- littlestymaar 2y ago
  You are moving the goalpost. The discussion has always been about transformers vs non transformers. You claimed that self attention was needed to achieve the level of intelligence that we've seen with GPT 3.5: > without those attention heads even the scaling up to current parameter sizes we have to day would not have ended up with the level of emergent intelligence that shocked the world with GPT 3.5. (Verbatim quote from you https://news.ycombinator.com/item?id=41986010 https://news.ycombinator.com/item?id=41986010) This is the claim I've been disputing, by responding that the key feature of the intelligence of tranformer models come from their scalability. And now that we have alternative that scale equally well (SSM and RWKV) unsurprisingly we see them achieve the same level of reasoning abilities. > Every statement above that I made, that you called wrong, was correct. lol. Well, except the one quoted above at least…
  - quantadev 2y ago
    In the quote you're calling wrong (41986010), you're interpreting "scaling up" as "scaling up, including changing architecture". Scaling up transformers just means scaling up transformers, and keeping everything else the same. In other words you're interpreting "parameter size" as "parameter size, independent of architecture", and I meant parameter size of a Transformer (in the context of with v.s. without Self-Attention).
    - littlestymaar 2y ago
      Pathetic.
      - quantadev 2y ago
        Straw-manning failed, so now you insult.
        littlestymaar 2y ago
        There's no staw-man, and you are now at the point of trying to re-invent the definition of words in order to somehow “win the argument ” without even respecting your own previous position. This behavior is legit pathetic, it's not an insult it's a fact. Respect yourself.
        quantadev 2y ago
        I stand by every word: 1) Self-Attention is more important than scale, and 2) to test that claim, simply remove SA from a transformer and see if it destroys the "intelligence" or not. There's nothing confusing about that, but thanks for your concerns and your polite words.
        littlestymaar 2y ago
        No that wasn't your argument and this new one is off course a much waker one that you fell back onto to be “technically right”. That attention heads are mandatory for transformers is a tautology (without it a transformer is just an MLP…) so of course this statement is going to be correct, by definition. But when you move the goal post to land on a tautology then you've surrendered your abilities to argue anything and you are just ridiculing yourself. Take this question of your for instance: > If you know of any models that have had success (even at the GPT-2 level) without Self-Attention, I'd be interested to know what they are, because I don't know of any. Which is a legit, non-ridiculous, one. If you replace it with your later much weaker argument: > > If you know of any MLP that have had success (even at the GPT-2 level), I'd be interested to know what they are, because I don't know of any. Then it becomes a dumb question given that MLP have no way of encoding context and can't process sequences of words in the first place. So when you argue that it was your argument all along, it's particularly embarrassing because you're just arguing that your previous arguments were equally dumb even when they weren't. That's why I said you're disrespecting your earlier argumentation by retreating to your later tautology.