3 ms·
Darn, I was hoping the RWKV people had finally obtained reportable results. This is still interesting, though. Maybe we will see more alternatives to transforme
by firejake308 3y ago
Darn, I was hoping the RWKV people had finally obtained reportable results. This is still interesting, though. Maybe we will see more alternatives to transformers soon
- refulgentis 3y agoThere are RWKV numbers in there, so you sort of got your wish sideways. :) Has RWKV not released anything until now? I thought it was an open project that was in use, if sort of by a hipster 1%
- stellaathena 3y agoPaper: https://arxiv.org/abs/2305.13048 https://arxiv.org/abs/2305.13048 Models (v4 are the ones from the paper): https://huggingface.co/RWKV https://huggingface.co/RWKV GitHub: https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM
- senseiV 3y agoV5 7b is out, close to hyena, gets 1400 t/s on a 3090, while an h100 llama 7b 8bit is 1200 t/s
- stellaathena 3y agoRWKV had a paper accepted at EMNLP and released models which match the performance of equivalent transformers. What else are you looking for?
- senseiV 3y agoThey Do, the latest rwkv v5, matches mamba at 3b scale, and from the benchmarks I see, its similar to hyena