25 ms·
> The Transformer models, used in this experiment, all have a single attention layer with model dimension and context length 32. I think we are going to need t
by lalaland1125 2y ago
> The Transformer models, used in this experiment, all have a single attention layer with model dimension and context length 32.
I think we are going to need to see more experiments here, especially because the theoretical motivations here are weak
- ttul 2y agoI’m certainly not a domain expert, but one thing I have read repeatedly about Transformers is that not all tricks scale the same.