4 ms·

I've been catching up on the papers, and the mechanics of RWKV are interesting, but I'm more of a hands on guy and I played a bunch with RWKV earlier this year

by lhl 3y ago

I've been catching up on the papers, and the mechanics of RWKV are interesting, but I'm more of a hands on guy and I played a bunch with RWKV earlier this year when its quality was comparable with the early LLaMA models of similar size and it was significantly faster (its CPU inference was comparable to the latter's GPU inference speeds 6 months ago), but I feel like those advantages have largely dissipated. With 4-bit quants, and the optimizations for batch=1 with llama.cpp, Exllama, MLC on the edge side, and on continuous batching w/ vLLM and the like on the high throughput side, I think performance (in terms of speed) is not as pressing, at least while people are pushing hardest on the capabilities front. Meanwhile, the quality of the best Llama 2 (and even the last-gen of Llama 1) models have blown by the existing RWKV models (although I'm definitely interested in kicking the tires on RWKV-5 when it's out). Also, I recently stepped back into RWKV world to check up on what was going on, and I was actually a bit shocked at how night and day the tooling/ecosystem is now. While it's not all roses the open LLM side, there's enough critical mass where the evolution has been ... extremely rapid, to say the least. I think over the next few years, obviously there will be architectural improvement that succeeds the current crop of dense transformer LLMs (one might argue that's already happening piece by piece), and I'm happy there are experiments like RWKV going on, so I guess we'll just have to see in terms of what scales and what doesn't.

pico_creator 3y ago
Quantisation thankfully is applicable to RWKV as much as transformers. Most notably in our RWKV.cpp community project: https://github.com/saharNooby/rwkv.cpp https://github.com/saharNooby/rwkv.cpp Tooling/Ecosystem is something that I am actively working on as there is still a gap to transformers level of tooling. But i'm glad that there is a noticeable difference! And yes! experiments are important, to ensure improvements in the architecture. Even if "Linear Transformers" replaces "Transformers". Alternatives should always be explored, to learn from such trade-offs to the benefit of the ecosystem (This was lightly covered in the podcast, where I share IMO that we should have more research into text based diffusion networks)
- lhl 3y ago
  Glad to hear that you're focused on improving tooling, as you point out, as there is definitely a gap (that maybe I was too circumspect about it originally, but to be clear, difference wasn't in RKWV's favor). Some of it may just be the docs. I think one easy improvement is that when you search for RWKV, you end up on the Github page, but the README basically goes from a description of the model, to a short snippet of sample inference code, to a list of implementations, to a bunch of random research notes? It's a bit scattered and pretty hard to read, but should be easy to clean up. From a practical perspective for example, there's no list/description in the repo or the model cards on all the different versions at https://huggingface.co/BlinkDL https://huggingface.co/BlinkDL (world, raven, pile, novel, pileplus, etc) and while it says RWKV-4-World is the "best" model, it's only a 7B and Raven 14B is listed above and says it's fine-tuned (while World is not)? Which is actually better? Unsure, no benchmarks (even the spreadsheet screenshots eventually listed don't give any info, nor do the model cards) - even as someone returning to RWKV this was very confusing. Compare it to what the Llama community is doing these days with model cards (eg https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-13B https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-... or for quants https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.0-GPTQ https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.0...) Also, for those not familiar with the project at all, I think a slightly expanded/better organized "Usage" section I think would be helpful in terms of getting up and running (it doesn't help that multiple implementations each have their own quant formats). I feel like have a section that would get a user easily up and running (I think ExLlama and MLC both do a good job here) would be helpful. Again, some of this might just be a pass of cleaning up/better organizing the docs.
  - pico_creator 3y ago
    Thanks for clarifying, and pointing out where docs can be improved. For most parts I am trying to ensure everything do get properly reorganised under the official wiki page here. I have just added a guideline of sort to help better navigate which model should be used accordingly, based on your feedback. https://wiki.rwkv.com/#which-rwkv-models-should-i-be-using https://wiki.rwkv.com/#which-rwkv-models-should-i-be-using Eventually, from an SEO stand-point, the first landing for RWKV should eventually be optimised to be the wiki or rwkv.com page itself. And not blinkDL original trainer. This would be more aligned with the fact that the vast majority of search for the model would naturally be individuals who would want to try the model (and not train it / finetune from scratch).

by lhl 3y ago

pico_creator 3y ago

Quantisation thankfully is applicable to RWKV as much as transformers. Most notably in our RWKV.cpp community project: https://github.com/saharNooby/rwkv.cpp https://github.com/saharNooby/rwkv.cpp Tooling/Ecosystem is something that I am actively working on as there is still a gap to transformers level of tooling. But i'm glad that there is a noticeable difference! And yes! experiments are important, to ensure improvements in the architecture. Even if "Linear Transformers" replaces "Transformers". Alternatives should always be explored, to learn from such trade-offs to the benefit of the ecosystem (This was lightly covered in the podcast, where I share IMO that we should have more research into text based diffusion networks)

lhl 3y ago
Glad to hear that you're focused on improving tooling, as you point out, as there is definitely a gap (that maybe I was too circumspect about it originally, but to be clear, difference wasn't in RKWV's favor). Some of it may just be the docs. I think one easy improvement is that when you search for RWKV, you end up on the Github page, but the README basically goes from a description of the model, to a short snippet of sample inference code, to a list of implementations, to a bunch of random research notes? It's a bit scattered and pretty hard to read, but should be easy to clean up. From a practical perspective for example, there's no list/description in the repo or the model cards on all the different versions at https://huggingface.co/BlinkDL https://huggingface.co/BlinkDL (world, raven, pile, novel, pileplus, etc) and while it says RWKV-4-World is the "best" model, it's only a 7B and Raven 14B is listed above and says it's fine-tuned (while World is not)? Which is actually better? Unsure, no benchmarks (even the spreadsheet screenshots eventually listed don't give any info, nor do the model cards) - even as someone returning to RWKV this was very confusing. Compare it to what the Llama community is doing these days with model cards (eg https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-13B https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-... or for quants https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.0-GPTQ https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.0...) Also, for those not familiar with the project at all, I think a slightly expanded/better organized "Usage" section I think would be helpful in terms of getting up and running (it doesn't help that multiple implementations each have their own quant formats). I feel like have a section that would get a user easily up and running (I think ExLlama and MLC both do a good job here) would be helpful. Again, some of this might just be a pass of cleaning up/better organizing the docs.
- pico_creator 3y ago
  Thanks for clarifying, and pointing out where docs can be improved. For most parts I am trying to ensure everything do get properly reorganised under the official wiki page here. I have just added a guideline of sort to help better navigate which model should be used accordingly, based on your feedback. https://wiki.rwkv.com/#which-rwkv-models-should-i-be-using https://wiki.rwkv.com/#which-rwkv-models-should-i-be-using Eventually, from an SEO stand-point, the first landing for RWKV should eventually be optimised to be the wiki or rwkv.com page itself. And not blinkDL original trainer. This would be more aligned with the fact that the vast majority of search for the model would naturally be individuals who would want to try the model (and not train it / finetune from scratch).