6 ms·
RWKV Language Model
- nullc 2y agoAnyone ever look at doing a MoE like composition with RWKV and a transformer?
- pico_creator 2y agoNot an MoE, but we have already done hybrid models. And found it to be highly performant (as per the training budget) https://arxiv.org/abs/2407.12077 https://arxiv.org/abs/2407.12077
- intalentive 2y agoIdea for killer app for recurrent models: low latency, low memory LLM / TTS coupling. Start decoding / generating speech as soon as new tokens are generated. When the LLM is cranking out token t, the TTS is already working on token t-1. It doesn’t have to wait. Then when the LLM is finished, the TTS is nearly finished too. The two models being colocated you just saved another network call as well. Recurrent models with constant hidden state are naturally suited to streaming data, potentially opening the door to unexplored new use cases.
- computerex 2y agoNew multimodal models take raw speech input and provide raw speech output, no tts in the middle.
- benob 2y agoA relatively detailed description of such systems: https://arxiv.org/abs/2410.00037 https://arxiv.org/abs/2410.00037
- Closi 2y agoSeems like the future - so much meaning and context is lost otherwise.
- intalentive 2y agoVery cool. Logical next step. Would be interested to know what the dataset looks like.
- moffkalast 2y agoYoutube. Youtube is the dataset.
- cootsnuck 2y agoThis can currently already be done using a streaming capable LLM with a streaming input/output TTS model.
- lostmsu 2y agoAny LLM is "streaming capable".
- Xmd5a 2y agohttps://github.com/mit-han-lab/streaming-llm https://github.com/mit-han-lab/streaming-llm On a side node, and that's what led me to the link above, I wonder if it would be possible to chain N streaming LLMs in an agent workflow and get a final output stream almost instantaneously without waiting for N-1 LLM to complete their reply.
- pico_creator 2y agoThis is actually the hypothesis for cartesia (state space team), and hence their deep focus on voice model specifically. Taking full advantage of recurrent models constant time compute, for low latencies. RWKV team's focus is still however is first in the multi-lingual text space, then multi-modal space in the future.
- swyx 2y agoKaran from Cartesia explains SSMs+voice really well: https://www.youtube.com/watch?v=U9DPRZ0lSIQ https://www.youtube.com/watch?v=U9DPRZ0lSIQ its one of those retrospectively obvious/genius insights that i wish i understood when i first met him
- yshui 2y agoAny autoregressive model can do what you are describing. transformers are generating one token at a time too, not all at once.
- whimsicalism 2y agoyes but transformers are much slower than state space models
- intalentive 2y agoTrue but memory requirements grow with sequence length. For recurrent models the memory requirement is constant. This is why I qualified with "low memory".
- sushidev 2y agoInteresting. Very cryptic for simple user like me. I wonder if it’s useful today and for what purposes
- pico_creator 2y agoCurrently the strongest RWKV model is 32B in size: https://substack.recursal.ai/p/q-rwkv-6-32b-instruct-preview https://substack.recursal.ai/p/q-rwkv-6-32b-instruct-preview This is a full drop in replacement for any transformer model use cases on model sizes 32B and under, as it has equal performance to existing open 32B models in most benchmarks We are in works on a 70B, which will be a full drop in replacement for most text use cases
- swyx 2y agohow about finetuning your 32B to be R1QWQKV?
- pico_creator 2y agoThere is a current lack of "O1 style" reasoning dataset in open source space. QWQ did not release their dataset. So that would take some time for the community to prepare. It's definitely something we are tracking to do as well =)
- lostmsu 2y agoWhy aren't you on lmarena (former chatbot arena) leaderboard?
- pico_creator 2y agokinda on a todo list, the model is open source on HF for anyone who is willing to make it work with lmarena
- swyx 2y agofor those who want a more conversational intro, we've been covering RWKV for a bit! 2023: https://latent.space/p/rwkv https://latent.space/p/rwkv 2024: https://www.youtube.com/watch?v=LPe6iC73lrc https://www.youtube.com/watch?v=LPe6iC73lrc <- offers a bit of professional compare and contrast vs state space models. i think its cool that now both RNN and LSTM (with xLSTM) now have modern attention-inspired variants that solve the previous issues. I wonder if 1) its possible to overcome the "hardware lottery" that transformers have now won, and 2) if recurrent/selective state can do the kind of proper lookback on extremely long context that we will want it to do to compete with full attention (easy to say no, harder to propose what to do about it). there's also Liquid AI, whatever it is that they do.
- inciampati 2y agothe recurrent model needs a mechanism to replay past context. no need to go quadratic to access all of it. they could replay multiple times to get effects similar to attention. the hardware lottery, well... imo it's really about leveraging fully parallel training to learn how to use a memory. attention is quadratic but it can be computed in parallel. it's an end to end learned memory. getting that kind of pattern into RNNs won't be easy but it's going to be crucial before we boil the ocean.
- pico_creator 2y agoRWKV already solve the parallel compute problem for GPU, based on the changes it has done - so it is a recurrent model that can scale to thousands++ of GPU no issue. Arguably with other recurrent architecture (State Space, etc) with very different design implementation. The issue of old recurrent design was just the way LSTM was designed. Not the recurrent nature.
- shawntan 2y agoAlthough marketed as such, RWKV isn't really an RNN. In the recent RWKV7 incarnation, you could argue it's a type of Linear RNN, but past versions had an issue of taking its previous state from a lower layer, allowing for parallelism, but makes it closer to a convolution than a recurrent computation. As for 1), I'd like to believe so, but it's hard to get people away from the addictive drug that is the easily parallelised transformer, 2) (actual) RNNs and attention mechanisms to me seem fairly powerful (expressivity wise) and perhaps most acceptable by the community.
- pico_creator 2y agoHey there, im Eugene / PicoCreator - co-leading the RWKV project - feel free to AMA =)
- theLiminator 2y agoDo you have an in depth comparison between RWKV and models like mamba or s4?
- pico_creator 2y agoNot sure how indepth you want it to be. But we did do a co-presentation with one of the coauthors of mamba at latent space : https://www.youtube.com/watch?v=LPe6iC73lrc https://www.youtube.com/watch?v=LPe6iC73lrc
- low_tech_punk 2y agoThanks! The 0.1B version looks perfect for embedded system. What is the key benefit of attention-free architecture?
- pico_creator 2y agolower compute cost especially over longer sequence length. Depending on context length, its 10x, 100x, or even 1000x+ cheaper. (quadratic vs linear cost difference)
- Ey7NFZ3P0nzAe 2y agoHas there been progress towards making RWKV multimodal? Can be use projector layers to send images to RWKV?
- pico_creator 2y agoThere is work done for Vision RWKV, and audio RWKV, an example paper is here: https://arxiv.org/abs/2403.02308 https://arxiv.org/abs/2403.02308 Its the same principle as open transformer models where an adapter is used to generate the embedding However currently the core team focus is in scaling the core text model, as this would be the key performance driver, before adapting multi-modal. The tech is there, the base model needs to be better
- smusamashah 2y agoHow does it compare with other LLMs in terms of performance? Is.it near GPT 3 or Llama or what?
- upghost 2y agoSeems really cool. Does anyone have any sample code to link to? Do RNN models use the same pytorch/hugging face Python stuff or is it completely different...?
- ianand 2y agoReminder that Microsoft ships RWKV with Windows (~1.5 billion devices), making it probably the most widely deployed non transformer model out there. Amazing work! https://blog.rwkv.com/p/rwkvcpp-shipping-to-half-a-billion https://blog.rwkv.com/p/rwkvcpp-shipping-to-half-a-billion ps Eugene you should brag about that on the homepage of RWKV.
- Fischgericht 2y ago"RWKV (pronounced RwaKuv)" - love it. How does the corw make? Rwa! Rwa! Rwa!
- bbor 2y agoThank god I’m not the only one stunned by that. I don’t need IPA, but this isn’t even vaguely pronounceable!
- bigattichouse 2y agoI've spent an afternoon attempting to compile and run RWKV7 locally.. and I just don't get it. lotta errors in compiling... and it's a lot. Like a lot, a lot... it's walls of versions and sub projects. Any kind of quickstart guide? Also found.tried rwkv.cpp, and I can't seem to compile that either.