3 ms·
To be sure, the seq2seq style already allowed multi length embeddings before transformers were a thing. RNN vs Transformer is entirely an implementation choice;
by sdenton4 3y ago
To be sure, the seq2seq style already allowed multi length embeddings before transformers were a thing. RNN vs Transformer is entirely an implementation choice; you can build a seq2seq model with any combination of transformer, RNN, and conventional layers.
Each of these layer types have different computational costs for training and inference, and encode different inductive biases, which may be more or less appropriate to a given problem.