4 ms·
> The architecture is pretty dumb, the secret is the training data If this were true, we could throw the same training data at any other "dumb" architecture an
by FartyMcFarter 3y ago
> The architecture is pretty dumb, the secret is the training data
If this were true, we could throw the same training data at any other "dumb" architecture and it would learn language at least as well/fast as transformers do. But we don't see that happening, so the architecture must be smartly designed for this purpose.
- visarga 3y agoActually there are alternatives by the hundreds, with similar results. Reformer, Linformer, Performer, Longformer... none is better than vanilla overall, they all have an edge in some use case. And then we have MLP-mixer which just doesn't do "attention" at all, MLP is all you need. A good solution for edge models.
- homarp 3y agoMLP-mixer: https://news.ycombinator.com/item?id=28581570 https://news.ycombinator.com/item?id=28581570 and https://towardsdatascience.com/mlp-mixer-in-a-nutshell-eccffb68e4fc https://towardsdatascience.com/mlp-mixer-in-a-nutshell-eccff...
- whimsicalism 3y agoOther dumb architectures don't parallelize as well. Other architectures that parallelize at similar levels (RNN-RWKV, H3, S4, etc.) do perform well at similar parameter counts and data sizes.
- homarp 3y agoRNN-RWKV - https://news.ycombinator.com/item?id=36038868 https://news.ycombinator.com/item?id=36038868 H3 - https://news.ycombinator.com/item?id=34673535 https://news.ycombinator.com/item?id=34673535 S4 - https://srush.github.io/annotated-s4/ https://srush.github.io/annotated-s4/