3 ms·
Currently the strongest RWKV model is 32B in size: https://substack.recursal.ai/p/q-rwkv-6-32b-instruct-preview https://substack.recursal.ai/p/q-rwkv-6-32b-inst
by pico_creator 2y ago
Currently the strongest RWKV model is 32B in size: https://substack.recursal.ai/p/q-rwkv-6-32b-instruct-preview https://substack.recursal.ai/p/q-rwkv-6-32b-instruct-preview
This is a full drop in replacement for any transformer model use cases on model sizes 32B and under, as it has equal performance to existing open 32B models in most benchmarks
We are in works on a 70B, which will be a full drop in replacement for most text use cases
- swyx 2y agohow about finetuning your 32B to be R1QWQKV?
- pico_creator 2y agoThere is a current lack of "O1 style" reasoning dataset in open source space. QWQ did not release their dataset. So that would take some time for the community to prepare. It's definitely something we are tracking to do as well =)
- lostmsu 2y agoWhy aren't you on lmarena (former chatbot arena) leaderboard?
- pico_creator 2y agokinda on a todo list, the model is open source on HF for anyone who is willing to make it work with lmarena