3 ms·
No. This is blatantly false. The belief that recurrent model can't be scaled is untrue. People have recently trained MAMBA with billions of parameters. The fund
by YetAnotherNick 3y ago
No. This is blatantly false. The belief that recurrent model can't be scaled is untrue. People have recently trained MAMBA with billions of parameters. The fundamental reason why transformers changed the field is that they are lot more scalable context length wise, and LSTM, LRU etc doesn't come close.
- HarHarVeryFunny 3y agoYes, but pure Mamba doesn't perform as well as a transformer (and neither did LTSMs). This is why you see hybrid architectures like Jamba = Mamba + transformer. The ability to attend to specific tokens is really key, and what is lost in recurrent models where sequence history is munged into a single state.
- YetAnotherNick 3y agoThat's my point. It doesn't perform in terms of loss, even though it performs well enough in terms of compute
- mjburgess 3y ago> they are lot more scalable context length wise Sure, we're agreeing. I'm just being less specific.
- YetAnotherNick 3y agoScalable as in loss wise scalable, not compute wise.