3 ms·
I'll struggle through the mamba papers, but does anyone have a good post or article that gives an intuition about what is fundamentally different about Mamba co
by marviel 3y ago
I'll struggle through the mamba papers, but does anyone have a good post or article that gives an intuition about what is fundamentally different about Mamba compared to transformers?
- TibbityFlanders 3y ago[dead]
- sevagh 3y agoI read this today: https://www.unite.ai/mamba-redefining-sequence-modeling-and-outforming-transformers-architecture/ https://www.unite.ai/mamba-redefining-sequence-modeling-and-...
- DarmokJalad1701 3y agoThis is an article about "S4" that uses "Structured State Spaces". https://srush.github.io/annotated-s4/ https://srush.github.io/annotated-s4/ The matrices that make up the state space (A, B and C) are constant in S4. This allowed them to represent some of the math operations as a convolution (which can be parallelized). The difference between S4 and Mamba is that these matrices are input-dependent in Mamba. Plus they add in some CUDA stuff ("parallel scan") to make it faster to compute on a GPU even if these matrices are not constant. Yannic Kilcher's video on Mamba might also be a good resource: https://youtu.be/9dSkvxS2EB0 https://youtu.be/9dSkvxS2EB0
- markrogersjr 3y agoI’d recommend starting with the appendix of the original hippo paper.
- RandyRanderson 3y agohttps://www.youtube.com/watch?v=9dSkvxS2EB0 https://www.youtube.com/watch?v=9dSkvxS2EB0 via https://www.youtube.com/@YannicKilcher https://www.youtube.com/@YannicKilcher
- anomaly72 3y agoWould recommend Sasha Rush's lecture https://www.youtube.com/watch?v=dKJEpOtVgXc https://www.youtube.com/watch?v=dKJEpOtVgXc, it provides the intuition for state space models via linear RNNs
- mochidusk 3y agoI struggled learning about Mamba's architecture but realized it's because I had some gaps in knowledge. In no particular order, they were: - a refresher on differential equations - legendre polynomials - state spaced models; you need to grok the essence of x' = Ax + Bu y = Cx - discretization of S4 - HiPPO matrix - GPU architecture (SRAM, HBM) Basically, transformers is an architecture that uses attention. Mamba is the same architecture that replaces attention with S4 - but this S4 is modified to overcome its shortcomings, allowing it to act like a CNN during training and an RNN during inference. I found this video very helpful: https://www.youtube.com/watch?v=8Q_tqwpTpVU https://www.youtube.com/watch?v=8Q_tqwpTpVU His other videos are really good too.
- fabmilo 3y agoI was about to post that video too. Highly recommended.
- ttul 3y agoI’ve spent a few hours reading the Mamba paper, examining the code, and watching Yannick’s YouTube dive into the paper. If you took discrete time signal processing in school, you will have what you need to work through the math. Here is my intuition about Mamba. It’s a linear discrete time signal filter where the filter blocks are conditioned on the input at each time step as well as receiving the input as a normal filter would. Think of the predecessor to Mamba as a discrete time filter without this conditioning. That’s the first innovation. The other innovation is some lovely algebra allowing them to reorganize things so they can be computed much more efficiently on current GPU hardware, with respect to slow and fast memory. State space models are very simple conceptually. It’s kind of amazing that Mamba does so well on long sequence modeling because the architecture is ridiculously simple. But when you consider that the state matrices are conditioned on the input, it all makes sense. Somehow, they learn how to adapt the “filter” based on the input. That’s where the intelligence lies.
- gschoeni 3y agoWe went over it in our Friday paper club before the holidays which helped me gain an intuition. https://blog.oxen.ai/mamba-linear-time-sequence-modeling-with-selective-state-spaces-arxiv-dives/ https://blog.oxen.ai/mamba-linear-time-sequence-modeling-wit... I'm still not convinced on Mamba's performance on Natural Language tasks, but maybe it's just because they haven't trained a large enough model on enough data yet.