3 ms·

I’ll bite: can anyone please eli5 to the non PhDs among us?

• cs702 2y ago
  TL;DR: The authors show that if you simplify Mamba so its state-space layer uses a diagonal matrix A that is a scalar times the identity matrix, the state-space transformation can be expressed as a form of causal linear attention.[a] That's the duality the authors refer to in the title. The key practical benefit is that it enables more efficient (faster) training on GPUs. --- [a] https://arxiv.org/abs/2006.16236 https://arxiv.org/abs/2006.16236
• brrrrrm 2y ago
  restricts freedom in one of the parameters (A) to make training substantially more efficient (easier for a GPU to churn through). the actual flops involved are similar to the original SSM-based version, but that's harder to formulate as strictly matrix multiplications
• xcodevn 2y ago
  tldr: mamba is not as good as transformer.
  • metalloid 2y ago
    Can you elaborate more?