4 ms·
As a (mostly) layperson, this seems like it could be a very significant paper. What are the odds we see the next few years of machine learning models based on R
by mindwok 3y ago
As a (mostly) layperson, this seems like it could be a very significant paper. What are the odds we see the next few years of machine learning models based on RWKV like we have seen with transformers since the attention is all you need paper?
- m3kw9 3y agoThis model seem to be good when input context is large in comparison to OpenAI, would just wait till if someone productize it. Most likely it wine be just one paper but many new and old ideas combined
- visarga 3y agoBeing a RNN there is another trick: caching a long prompt, because RNNs only look back one step while transformers see the whole sequence. So you can load your long context only once and reuse it many times.
- pico_creator 3y agoYup. This is commonly done in the community for the chat models as well (due to the huge amount of reuse for each reply)
- FallDead 3y agoPretty high, we are going to be deploying something similar to Sono's GPT based LLM, and leverage RWKV for that, still playing around with it though!