4 ms·
This paper introduces a new class of models called selective state space models (S6 or selective SSMs). The key ideas and results are: 1. SSMs are a type of re
by ttul 3y ago
This paper introduces a new class of models called selective state space models (S6 or selective SSMs). The key ideas and results are:
1. SSMs are a type of recurrent model that can scale linearly with sequence length, making them more efficient than Transformers. However, prior SSMs struggled with discrete sequence modeling tasks like language.
2. This paper augments SSMs with a "selection mechanism" that allows the model dynamics to depend on the input, giving it the ability to selectively remember or forget information. This makes SSMs effective on tasks requiring discrete reasoning.
3. They design an efficient parallel scan algorithm to implement selective SSMs on GPUs. Despite recurrency, this achieves up to 5x higher throughput over Transformers in benchmarks.
4. They simplify prior SSM architectures into a new model called Mamba. On language modeling, Mamba matches or exceeds Transformers of 2-3x its size, while retaining linear scaling. It also achieves state-of-the-art results on audio, genomics, and synthetic tasks requiring long-term reasoning.
This work makes SSMs truly competitive with Transformers through selectivity and efficient engineering. Mamba matches or beats Transformers on major modalities while being substantially more efficient in computation, memory, and scaling to long sequences. If replicated, it's arguably the first linear-time architecture with Transformer-quality performance!!
Can't wait to see the code!
- deleted 3y ago[deleted]
- qumpis 3y agoHow does this differ from RNNs and their gating mechanism?
- eli_gottlieb 3y agoAm I understanding correctly that Mamba SSM's can be used for autoregressive sequence modeling, on both continuous and discrete data?
- ttul 3y agoThe paper claims that Mamba outperforms Transformers on modeling audio waveforms. Once a signal gets into a computer, it is technically always discrete, but I think what you’re referring to here is the ability to model continuous signals like audio and discrete data such as written language. Am I reading you right?
- eli_gottlieb 3y agoYes I am indeed talking about that distinction.
- triyambakam 3y agoI got a really similar summary using the paper abstract in context window with prompt to summarize with GPT4. Did you write that summary yourself?
- ttul 3y agoI got Claude to write a draft and then I scanned the paper to make sure it didn’t make any egregious errors. I cleaned up its output, removing some bullets that I felt didn’t add anything the abstract had already covered well. I also prompted it to contrast this development with earlier work.
- smaddox 3y ago> If replicated, it's arguably the first linear-time architecture with Transformer-quality performance!! RetNet was published in August (although I only learned of it last week): https://arxiv.org/abs/2307.08621 https://arxiv.org/abs/2307.08621
- saberience 3y agoIf you’re going to use ChatGPT to write comments you should preface the comment saying so. I don’t want Hackernews to be a place where people just copy and paste ChatGPT work, taking credit for it, and also potentially adding hallucinations to the ecosystem here. When I read comments I assume they are human written, that someone put effort into that comment, and when someone is strongly asserting something (as your comment does), that they have sufficient authority to match the strength of the assertion. ChatGPT comments always sound VERY sure of themselves, as though they are 100% right, but I know from experience that it can be very wrong, especially with more complex topics. So I think it’s poor form to post up this kind of comment with its strong assertions without saying you did a copy and paste as though it was your own original work. A modern version of plagiarism really while also lowering the overall quality of Hackernews and polluting it with generated text.
- aix1 3y agoIn fairness to the GP, there's another comment from them from about an hour ago explaining how the answer was produced. tl;dr Claude drafting + manual checking and cleanup.
- ttul 3y agoYeah, as the other comment here indicates, I used Claude to generate an initial summary, which I verified and edited before posting. I am teaching myself deep learning and I use Claude to analyze papers before diving in. Many papers are hard to get through otherwise - and that’s true I think even if you are an expert!
- mistymountains 3y agoYou won’t teach yourself by running papers through Claude, and you won’t need to if you went from first principles rather than rushing.
- ttul 3y agoI beg to differ. If I start with a summary that was produced by a model that has an excellent broad understanding of all prior work, it can help to fill in gaps so that when I do read the paper later, I parse it better. I have ADHD and a learning disability. I guess your mileage may vary.