3 ms·
> I don't expect them to replace transformers until they make some hypothetic breakthrough Yes, a breakthrough that does what Self-Attention is doing, rather t
by quantadev 2y ago
> I don't expect them to replace transformers until they make some hypothetic breakthrough
Yes, a breakthrough that does what Self-Attention is doing, rather than just scaling up.
- littlestymaar 2y agoNo, that's what you're missing from the beginning: the breakthrough of transformers was scalability. Now we have other models that are equally scalable and as such roughly equally performant (and that's not a surprise). But the ship has sailed and nobody is gonna switch to something else than transformers if it's not significantly better, and as such the other approaches are going to stay behind because every marginal improvement come to transformers first (because that's what practically everyone is working on) and alternative models are playing catch-up. This is a remarkable example of path dependence. Interpreting this as “transformers are fundamentally superior” is the mistake I'm trying to help you correct. The breakthrough of transformers was scalability. The next breakthrough of equivalent importance will be entirely different or it won't be.
- quantadev 2y agoScalability wasn't an architectural breakthrough. It was merely a discovery..
- littlestymaar 2y agoHow are these words even in contradiction to each other?
- quantadev 2y agoBy intentionally lacking context.
- littlestymaar 2y agoYou aren't even trying to pretend your sentence make sense, I see…
- quantadev 2y agoso Kindergarten bro.
- littlestymaar 2y agoWe are now at the Markov chain level of sentence, it's not even grammatically resembling a sentence at that point. Congratz!
- quantadev 2y agoLike your 1 word replies.
- littlestymaar 2y agoNo.
- deleted 2y ago[deleted]
- quantadev 2y agoYour post attacking my grammar contains two grammatical errors of it's own. Can you spot them?
- littlestymaar 2y agoI'm not attacking your grammar, but now I question your ability to understand a sentence. (English isn't my first language, BTW, so I'd be grateful if you could point the grammar errors)