3 ms·
I'm glad we're seeing exploration into scaling post-transformer LLM architectures, but I'm disappointed that it has a context window. That was kind of the selli
by kelseyfrog 3y ago
I'm glad we're seeing exploration into scaling post-transformer LLM architectures, but I'm disappointed that it has a context window. That was kind of the selling point of Mamba(and SSM models in general), right linear scaling because state+input=next_state+output?
- refulgentis 3y agoI'm not sure I follow fully, it is also the case for (handwaves) "traditional" LLMs that state + input = next state + output. Its just that output increases, so as output becomes input, eventually state + input / next state + output is greater than the context size. Re: linear scaling, that means the runtime cost is O(n) to context size, rather than traditional transformer O(n^2)
- maccam912 3y agoI think kelseyfrog meant that the state for a mamba model is supposed to "remember" stuff even if it doesn't have the actual tokens to reference any more. It might not be guaranteed to hang on to some information about tokens from a long time ago, but at least in theory it's possible, whereas tokens from before a context window in a tradional llms may as well never have existed.
- kelseyfrog 3y agoYes, you said it better than I did :)
- visarga 3y agoThat is valid for Mamba, this model (Jamba) is a mix of transformer and mamba layers, so it still has a quadratic memory cost, but divided by 8.
- a_wild_dandan 3y agostate = context The difference between SSMs and GPTs here is how that state/context scales. Per usual in engineering, there are big trade offs!
- kelseyfrog 3y agoI'm not following. State is a multi-dimensional vector and context is a list of tokens. State is perturbed by A and Bx(t), while context is appended to by sampling the predicted token distribution.
- spxneo 3y ago256k is huge dude. that is like 1/2 of the average non fiction novel i think at least 200~300 pages of PDF im not complaining here and it also fits in GPU