3 ms·

Mamba is the new and hot "linear transformer", could one day replace GPT based LLMs and scale sequence length up to 1M. It uses a clever math trick to paralleli

• erichocean 3y ago
  > while keeping a constant state size for generating the output Isn't Mamba's attention mechanism linear relative to the input? The innovation as I understand it is that attention in Mamba *isn't* quadratic (like transformers).
  • junipertea 3y ago
    State size is constant (a matrix), attention is linear (a single unidirectional scan over all states) Both statements are correct
  • prefer_to_ask 3y ago
    The innovation is that they managed to write a hardware-aware kernel to make it run fast/efficiently on GPUs. The authors of Mamba are the same authors of FlashAttention which was a performance optimization kernel written to reduce IO while computing normal O(N^2) attention. As I understand it, previously SSM models were not as easily parallelizable as the Transformer architecture.
    • kelseyfrog 3y ago
      Yes and no. There's an dual connection between ssms and convolutional models if certain constraints are met. Training convolutionally and inferring sequentially seeks a compromise between the two sides. I think we're about to find out the degree to which those constraints impact "easily".
    • wityl 3y ago
      They were previously parallelizable (via fft), but performed poorly on language modeling tasks. Mamba adds a dependence on the inputs that makes language modeling competitive with transformers, but that prevents using the fft approach. So they switch to a method using parallel prefix scan.