3 ms·
Too new is definitely one thing. Someone is going to have to make a gamble to actually paying for a serious pretraining run with this architecture before we kno
by thatguysaguy 3y ago
Too new is definitely one thing. Someone is going to have to make a gamble to actually paying for a serious pretraining run with this architecture before we know how it really stacks up against transformers.
There are some papers suggesting that transformers are better than SSMs in fundamental ways (e.g. They cannot do arbitrary key-based recall from their context: https://arxiv.org/abs/2402.01032 https://arxiv.org/abs/2402.01032). This means it's not just a no-brainer to switch over.
- gaogao 3y agoIt's a reasonably easy bet that Together is doing or will do a serious pretraining run with Mamba, where if that's a success other players might start considering it more.
- whimsicalism 3y ago> There are some papers suggesting that transformers are better than SSMs in fundamental ways I mean the vanilla transformers are also shown failing at the tasks they present.
- espadrine 3y agoAnother element is that Mamba required a very custom implementation down to custom fused kernels which I expect would need to be implemented in deepspeed or the equivalent library for a larger training run spanning thousands of GPUs.
- cs702 3y agoNot necessarily: https://www.reddit.com/r/MachineLearning/comments/1amb3xu/d_mamba_with_cumulative_sums/ https://www.reddit.com/r/MachineLearning/comments/1amb3xu/d_...