5 ms·

Worringly, I am it sure the people working on this really understand what a Transformer is Quote from them: “ There is still active research in non-transform

by zwaps 3y ago

Worringly, I am it sure the people working on this really understand what a Transformer is Quote from them: “ There is still active research in non-transformer based language models though, such as Amazon’s AlexaTM 20B which outperforms GPT-3“ Quote from said paper “ For AlexaTM 20B, we used the standard Transformer model architecture“ (Its just an encoder decoder transformer)

RC_ITR 3y ago
Yeah, I think the (worrying) confusion is that Amazon calls it a seq2seq model, which was the name of a SOTA RNN from Google a while back. Ofc now, seq2seq just means what you said (an encoder/decoder model, which is actually what a “truly vanilla” transformer would be anyway). The fact that any serious researcher thinks any other serious researchers are using models without self attention is the real red flag here. No one is trying to use other models anymore because they do not scale. There’s enough variety within transformers that you could argue we need a new level of taxonomy, but transformers are basically it for now.
- abidlabs 3y ago
  That's actually not true: https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM
  - RC_ITR 3y ago
    My understanding of both this and Apple AFT is that they are trained with attention, but then inference is done as an RNN. Is your understanding different?
sgk284 3y ago
Thanks for pointing this out. That was my mistake – my brain must have swapped out "different transformer architectures" with "different model architectures". I just updated the guide: https://github.com/brexhq/prompt-engineering/commit/3a3ac17a89f3929edae865f2933eade26c63bc28 https://github.com/brexhq/prompt-engineering/commit/3a3ac17a...
- abidlabs 3y ago
  A better example would have been the RWKV [1] [1] https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM

by zwaps 3y ago

RC_ITR 3y ago

Yeah, I think the (worrying) confusion is that Amazon calls it a seq2seq model, which was the name of a SOTA RNN from Google a while back. Ofc now, seq2seq just means what you said (an encoder/decoder model, which is actually what a “truly vanilla” transformer would be anyway). The fact that any serious researcher thinks any other serious researchers are using models without self attention is the real red flag here. No one is trying to use other models anymore because they do not scale. There’s enough variety within transformers that you could argue we need a new level of taxonomy, but transformers are basically it for now.

abidlabs 3y ago
That's actually not true: https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM
- RC_ITR 3y ago
  My understanding of both this and Apple AFT is that they are trained with attention, but then inference is done as an RNN. Is your understanding different?

sgk284 3y ago

Thanks for pointing this out. That was my mistake – my brain must have swapped out "different transformer architectures" with "different model architectures". I just updated the guide: https://github.com/brexhq/prompt-engineering/commit/3a3ac17a89f3929edae865f2933eade26c63bc28 https://github.com/brexhq/prompt-engineering/commit/3a3ac17a...

abidlabs 3y ago
A better example would have been the RWKV [1] [1] https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM