4 ms·
It’s going to take time. I can’t speak to the actual quality of mamba other than to say the authors are extraordinary and should be taken seriously. But traini
by solidasparagus 2y ago
It’s going to take time. I can’t speak to the actual quality of mamba other than to say the authors are extraordinary and should be taken seriously.
But training a large model requires a huge amount of capital so the biggest runs are designed around risk minimization. And remember, many of the decision makers of these runs made are in their positions by doing transformer-centric work. The true value of mamba is still unclear to me with very long context techniques being effective for transformers.
- inciampati 2y agoTo be frank, the long-context techniques that you're describing are still extremely limited in the length of context to consider, only a million-token order, and extremely expensive to apply.
- imtringued 2y agoAnd yet quadratic attention is still unavoidable. Anything that lets you have sub quadratic attention is going to have an accuracy Vs performance tradeoff.
- inciampati 2y agoNo, you can make multiple passes across the data set. There's no reason that you have to use quadratic attention. I point to the shape of our own brains, which are limited in size. Yet, through multiple passes over information and random access, which we guide, we're able to process very large systems.