4 ms·

Question: I only ever hear how RWKV is threatening the dominance of transformers... from the RWKV folks, while everyone independent is happily using transformer

Question: I only ever hear how RWKV is threatening the dominance of transformers... from the RWKV folks, while everyone independent is happily using transformers. Is there any substance to the above claim that's been independently verified? Not an NLP person, so apologies in advance if I've missed any obvious papers or references.

ianbutler 3y ago
Personally, the RNN "mode" runs substantially faster with similar quality to other models of the same size. RWKV is limited by compute and thus model size at present which makes it hard to suggest how it will stack up against models like falcon and llama2 in their larger sizes. There are also trade offs around their specific attention mechanism. It's approximate and consists of an exponential decay across time steps and some constant that represents the magnitude of the attention at that time step. The decay means for long running contexts it in theory will loses some of the ability to recall nuance you see with perfect attention on longer sequences. This isn't a specific problem of RWKV, but really any decay based attention. A perk of the formulation is the attention computation is linear in time complexity and only requires one additional serial scan at the end, so you get nicer scaling properties than quadratic perfect attention for the trade off in lossy attention. The examples in the original paper don't show this to be a problem, but they also show much shorter sequences. In the real world its a toss up, I'd expect it to be faster but more lossy for longer context windows. TLDR: It has some teeth, but I think saying it's threatening is wishful thinking to some degree. Good model, looking forward to seeing larger sizes.
- pico_creator 3y ago
  Agreed that as a whole, we do need more public benchmarks (both transformers and RWKV), with much longer sequence length. I view this as a case of the "benchmarks" and "datasets" not keeping up with the current progress, as the vast majority of benchmarks presumes the limit of up to 2k tokens. Which was cutting edge last year, but isn't today. Internally we do have some benchmarking for how the decay over time affects lossy memory, and we are making major improvements in this direction for v5, and aim for no loss within the 8k to 100k and beyond context sizes. And hopefully with time, there will be more formalised standard benchmarks for larger context sizes. Edit/PS: I believe until we get a sponsor for the compute required to build on similar scale and dataset compared to LLaMA2, so that we can have that "showdown" - we would be stuck - in its non threatening status. As there will always be that shadow of doubt, if it can scale to the next milestone that transformers have already crossed.
NLPaep 3y ago
RWKV performs poorly in practice.
- capableweb 3y ago
  You'd like to expand that a little, suggestively how you know this? Or you're expecting rebuttals that amount to "No, it doesn't"?
  - allisdust 3y ago
    Occam's razor tells us that if it's a great architecture/technical breakthrough, it would have taken the world by storm by now. Similar to the original transformer paper and model. Since 2017, the only successful models are variations of transformers. RNNs are no where in the picture. Simply believing a architecture is superior doesn't make it so. Nothing converges and performs as good as a model with attention in both training and inference. The difference is night and day.
    - capableweb 3y ago
      > Occam's razor tells us that if it's a great architecture/technical breakthrough, it would have taken the world by storm by now. Similar to the original transformer paper and model. I think you're making the opposite case here without even realizing it yourself. The Transformer was proposed in 2017, with the softmax-based attention being proposed in 2014 but it wasn't until 2023 when GPT-3 et al took the world by storm that people started really using the Transformer as it is today. The timeline going from being proposed to being used in production models also took multiple steps, with things improving with each step. It's a iterative process, not "dump -> done". > Simply believing a architecture is superior doesn't make it so I agree. But also I agree the opposite, that just because you believe it's not superior, doesn't make it so. If you claim "RWKV performs poorly in practice", I expect you to have something to back that up, definitely more than "Just because you believe it's superior doesn't make it so"
      - ianbutler 3y ago
emadm 3y ago
It has good potential and nobody ever thought that RNN's would be able to scale as they have led by BlinkDL. Well we did at Stability AI which is why we provided and scaled compute for it. They have the potential for far better edge inference and also benefits longer context windows. There are other architectures out there too we are supporting, I would be surprised if the current transformer LLM paradigm is what we use in a few years.
- polygamous_bat 3y ago
  With all due respect, I consider you as one of the "RWKV folks" I mentioned in the initial comment :) (assuming you are Emad Mostaque?) > They have the potential for far better edge inference and also benefits longer context windows. ... I would be surprised if the current transformer LLM paradigm is what we use in a few years. What are the studies/evidence that convinced you? Given the current landscape, the last claim is quite bold.
- pico_creator 3y ago
  What other architectures should we be taking note of (that stability is supporting)? Especially in the text space that you feel will change the current transformer LLM paradigm?
lhl 3y ago
I've been catching up on the papers, and the mechanics of RWKV are interesting, but I'm more of a hands on guy and I played a bunch with RWKV earlier this year when its quality was comparable with the early LLaMA models of similar size and it was significantly faster (its CPU inference was comparable to the latter's GPU inference speeds 6 months ago), but I feel like those advantages have largely dissipated. With 4-bit quants, and the optimizations for batch=1 with llama.cpp, Exllama, MLC on the edge side, and on continuous batching w/ vLLM and the like on the high throughput side, I think performance (in terms of speed) is not as pressing, at least while people are pushing hardest on the capabilities front. Meanwhile, the quality of the best Llama 2 (and even the last-gen of Llama 1) models have blown by the existing RWKV models (although I'm definitely interested in kicking the tires on RWKV-5 when it's out). Also, I recently stepped back into RWKV world to check up on what was going on, and I was actually a bit shocked at how night and day the tooling/ecosystem is now. While it's not all roses the open LLM side, there's enough critical mass where the evolution has been ... extremely rapid, to say the least. I think over the next few years, obviously there will be architectural improvement that succeeds the current crop of dense transformer LLMs (one might argue that's already happening piece by piece), and I'm happy there are experiments like RWKV going on, so I guess we'll just have to see in terms of what scales and what doesn't.
- pico_creator 3y ago
  Quantisation thankfully is applicable to RWKV as much as transformers. Most notably in our RWKV.cpp community project: https://github.com/saharNooby/rwkv.cpp https://github.com/saharNooby/rwkv.cpp Tooling/Ecosystem is something that I am actively working on as there is still a gap to transformers level of tooling. But i'm glad that there is a noticeable difference! And yes! experiments are important, to ensure improvements in the architecture. Even if "Linear Transformers" replaces "Transformers". Alternatives should always be explored, to learn from such trade-offs to the benefit of the ecosystem (This was lightly covered in the podcast, where I share IMO that we should have more research into text based diffusion networks)
  - lhl 3y ago
    Glad to hear that you're focused on improving tooling, as you point out, as there is definitely a gap (that maybe I was too circumspect about it originally, but to be clear, difference wasn't in RKWV's favor). Some of it may just be the docs. I think one easy improvement is that when you search for RWKV, you end up on the Github page, but the README basically goes from a description of the model, to a short snippet of sample inference code, to a list of implementations, to a bunch of random research notes? It's a bit scattered and pretty hard to read, but should be easy to clean up. From a practical perspective for example, there's no list/description in the repo or the model cards on all the different versions at https://huggingface.co/BlinkDL https://huggingface.co/BlinkDL (world, raven, pile, novel, pileplus, etc) and while it says RWKV-4-World is the "best" model, it's only a 7B and Raven 14B is listed above and says it's fine-tuned (while World is not)? Which is actually better? Unsure, no benchmarks (even the spreadsheet screenshots eventually listed don't give any info, nor do the model cards) - even as someone returning to RWKV this was very confusing. Compare it to what the Llama community is doing these days with model cards (eg https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-13B https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-... or for quants https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.0-GPTQ https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.0...) Also, for those not familiar with the project at all, I think a slightly expanded/better organized "Usage" section I think would be helpful in terms of getting up and running (it doesn't help that multiple implementations each have their own quant formats). I feel like have a section that would get a user easily up and running (I think ExLlama and MLC both do a good job here) would be helpful. Again, some of this might just be a pass of cleaning up/better organizing the docs.
swyx 3y ago
(interviewer here, but not on RWKV team and no ties) so if you're looking for social proof - this is something we cover in the pod but the original idea came from Apple (Attention Free Transformers) and Microsoft recently put out the RetNet which is a similar idea. If you believe that the reasoning benchmarks shown were legit (and these are pretty standard benchmarks imo, and audited by other academics not strictly on the RWKV team like Quentin Anthony (also our past guest!), tho he is with Eleuther), then that's pretty much all the proof you need. plus the fact that you can spin it up on huggingface spaces right now and try it out yourself (link in the show notes). bottomline is this thing is being built by the community that uses it - there is no higher form of dogfooding. if you wanna wait 3-4 years for big names to use it before you take it seriously, then we'll just have to wait. i call this out explictly on the pod intro that i'm trying to be early here, i think we might be the first big ai pod to seriously cover RWKV
pico_creator 3y ago
It is important to note the claims are made for similar dataset & params. If you like some independent verification outside eleuther circle. There is microsoft paper on retnet: https://arxiv.org/abs/2307.08621 https://arxiv.org/abs/2307.08621 And the stanford heyena paper: https://arxiv.org/abs/2302.10866 https://arxiv.org/abs/2302.10866 Do note that both cited are competing alternatives for linear transformers, and in some sense competes with RWKV, in this upcoming "linear transfromer architecture" trend. --- While all the above validates architecture. It's important to separate out architecture design from dataset. The quality of an AI model answers, is equal parts as much about the dataset, than the architecture used. This is why it makes less sense comparing LLaMA2 directly with retnet/RWKV for example, due to the large differences in dataset size. > Disclaimer: I am the same Eugene in the podcast
pico_creator 3y ago
Btw, I also realise on rereading, both statements can be true at the same time. In the podcast as well, I was being very transparent that I am happily using transformers (Salesforce codegen) in production for my primary work use cases, and I am still happily doing so. My work into RWKV is to support a new usecase which transformer could not support even today. And in no means should it threaten your happiness factor in using existing AI models. I suspect in the long run, AI architecture wars may slowly mirror that of the database wars, in software development (SQL vs NoSQL vs ??) Each of them will have substantial benefits in one use case over another. And may have many overlaps, with lots of happy users for all of them =)

ianbutler 3y ago

Personally, the RNN "mode" runs substantially faster with similar quality to other models of the same size. RWKV is limited by compute and thus model size at present which makes it hard to suggest how it will stack up against models like falcon and llama2 in their larger sizes. There are also trade offs around their specific attention mechanism. It's approximate and consists of an exponential decay across time steps and some constant that represents the magnitude of the attention at that time step. The decay means for long running contexts it in theory will loses some of the ability to recall nuance you see with perfect attention on longer sequences. This isn't a specific problem of RWKV, but really any decay based attention. A perk of the formulation is the attention computation is linear in time complexity and only requires one additional serial scan at the end, so you get nicer scaling properties than quadratic perfect attention for the trade off in lossy attention. The examples in the original paper don't show this to be a problem, but they also show much shorter sequences. In the real world its a toss up, I'd expect it to be faster but more lossy for longer context windows. TLDR: It has some teeth, but I think saying it's threatening is wishful thinking to some degree. Good model, looking forward to seeing larger sizes.

pico_creator 3y ago
Agreed that as a whole, we do need more public benchmarks (both transformers and RWKV), with much longer sequence length. I view this as a case of the "benchmarks" and "datasets" not keeping up with the current progress, as the vast majority of benchmarks presumes the limit of up to 2k tokens. Which was cutting edge last year, but isn't today. Internally we do have some benchmarking for how the decay over time affects lossy memory, and we are making major improvements in this direction for v5, and aim for no loss within the 8k to 100k and beyond context sizes. And hopefully with time, there will be more formalised standard benchmarks for larger context sizes. Edit/PS: I believe until we get a sponsor for the compute required to build on similar scale and dataset compared to LLaMA2, so that we can have that "showdown" - we would be stuck - in its non threatening status. As there will always be that shadow of doubt, if it can scale to the next milestone that transformers have already crossed.

NLPaep 3y ago

RWKV performs poorly in practice.

capableweb 3y ago
You'd like to expand that a little, suggestively how you know this? Or you're expecting rebuttals that amount to "No, it doesn't"?
- allisdust 3y ago
  Occam's razor tells us that if it's a great architecture/technical breakthrough, it would have taken the world by storm by now. Similar to the original transformer paper and model. Since 2017, the only successful models are variations of transformers. RNNs are no where in the picture. Simply believing a architecture is superior doesn't make it so. Nothing converges and performs as good as a model with attention in both training and inference. The difference is night and day.
  - capableweb 3y ago
    > Occam's razor tells us that if it's a great architecture/technical breakthrough, it would have taken the world by storm by now. Similar to the original transformer paper and model. I think you're making the opposite case here without even realizing it yourself. The Transformer was proposed in 2017, with the softmax-based attention being proposed in 2014 but it wasn't until 2023 when GPT-3 et al took the world by storm that people started really using the Transformer as it is today. The timeline going from being proposed to being used in production models also took multiple steps, with things improving with each step. It's a iterative process, not "dump -> done". > Simply believing a architecture is superior doesn't make it so I agree. But also I agree the opposite, that just because you believe it's not superior, doesn't make it so. If you claim "RWKV performs poorly in practice", I expect you to have something to back that up, definitely more than "Just because you believe it's superior doesn't make it so"
    - ianbutler 3y ago

emadm 3y ago

It has good potential and nobody ever thought that RNN's would be able to scale as they have led by BlinkDL. Well we did at Stability AI which is why we provided and scaled compute for it. They have the potential for far better edge inference and also benefits longer context windows. There are other architectures out there too we are supporting, I would be surprised if the current transformer LLM paradigm is what we use in a few years.

polygamous_bat 3y ago
With all due respect, I consider you as one of the "RWKV folks" I mentioned in the initial comment :) (assuming you are Emad Mostaque?) > They have the potential for far better edge inference and also benefits longer context windows. ... I would be surprised if the current transformer LLM paradigm is what we use in a few years. What are the studies/evidence that convinced you? Given the current landscape, the last claim is quite bold.
pico_creator 3y ago
What other architectures should we be taking note of (that stability is supporting)? Especially in the text space that you feel will change the current transformer LLM paradigm?

lhl 3y ago

I've been catching up on the papers, and the mechanics of RWKV are interesting, but I'm more of a hands on guy and I played a bunch with RWKV earlier this year when its quality was comparable with the early LLaMA models of similar size and it was significantly faster (its CPU inference was comparable to the latter's GPU inference speeds 6 months ago), but I feel like those advantages have largely dissipated. With 4-bit quants, and the optimizations for batch=1 with llama.cpp, Exllama, MLC on the edge side, and on continuous batching w/ vLLM and the like on the high throughput side, I think performance (in terms of speed) is not as pressing, at least while people are pushing hardest on the capabilities front. Meanwhile, the quality of the best Llama 2 (and even the last-gen of Llama 1) models have blown by the existing RWKV models (although I'm definitely interested in kicking the tires on RWKV-5 when it's out). Also, I recently stepped back into RWKV world to check up on what was going on, and I was actually a bit shocked at how night and day the tooling/ecosystem is now. While it's not all roses the open LLM side, there's enough critical mass where the evolution has been ... extremely rapid, to say the least. I think over the next few years, obviously there will be architectural improvement that succeeds the current crop of dense transformer LLMs (one might argue that's already happening piece by piece), and I'm happy there are experiments like RWKV going on, so I guess we'll just have to see in terms of what scales and what doesn't.

pico_creator 3y ago
Quantisation thankfully is applicable to RWKV as much as transformers. Most notably in our RWKV.cpp community project: https://github.com/saharNooby/rwkv.cpp https://github.com/saharNooby/rwkv.cpp Tooling/Ecosystem is something that I am actively working on as there is still a gap to transformers level of tooling. But i'm glad that there is a noticeable difference! And yes! experiments are important, to ensure improvements in the architecture. Even if "Linear Transformers" replaces "Transformers". Alternatives should always be explored, to learn from such trade-offs to the benefit of the ecosystem (This was lightly covered in the podcast, where I share IMO that we should have more research into text based diffusion networks)
- lhl 3y ago
  Glad to hear that you're focused on improving tooling, as you point out, as there is definitely a gap (that maybe I was too circumspect about it originally, but to be clear, difference wasn't in RKWV's favor). Some of it may just be the docs. I think one easy improvement is that when you search for RWKV, you end up on the Github page, but the README basically goes from a description of the model, to a short snippet of sample inference code, to a list of implementations, to a bunch of random research notes? It's a bit scattered and pretty hard to read, but should be easy to clean up. From a practical perspective for example, there's no list/description in the repo or the model cards on all the different versions at https://huggingface.co/BlinkDL https://huggingface.co/BlinkDL (world, raven, pile, novel, pileplus, etc) and while it says RWKV-4-World is the "best" model, it's only a 7B and Raven 14B is listed above and says it's fine-tuned (while World is not)? Which is actually better? Unsure, no benchmarks (even the spreadsheet screenshots eventually listed don't give any info, nor do the model cards) - even as someone returning to RWKV this was very confusing. Compare it to what the Llama community is doing these days with model cards (eg https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-13B https://huggingface.co/Open-Orca/OpenOrcaxOpenChat-Preview2-... or for quants https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.0-GPTQ https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.0...) Also, for those not familiar with the project at all, I think a slightly expanded/better organized "Usage" section I think would be helpful in terms of getting up and running (it doesn't help that multiple implementations each have their own quant formats). I feel like have a section that would get a user easily up and running (I think ExLlama and MLC both do a good job here) would be helpful. Again, some of this might just be a pass of cleaning up/better organizing the docs.

swyx 3y ago

(interviewer here, but not on RWKV team and no ties) so if you're looking for social proof - this is something we cover in the pod but the original idea came from Apple (Attention Free Transformers) and Microsoft recently put out the RetNet which is a similar idea. If you believe that the reasoning benchmarks shown were legit (and these are pretty standard benchmarks imo, and audited by other academics not strictly on the RWKV team like Quentin Anthony (also our past guest!), tho he is with Eleuther), then that's pretty much all the proof you need. plus the fact that you can spin it up on huggingface spaces right now and try it out yourself (link in the show notes). bottomline is this thing is being built by the community that uses it - there is no higher form of dogfooding. if you wanna wait 3-4 years for big names to use it before you take it seriously, then we'll just have to wait. i call this out explictly on the pod intro that i'm trying to be early here, i think we might be the first big ai pod to seriously cover RWKV

pico_creator 3y ago

It is important to note the claims are made for similar dataset & params. If you like some independent verification outside eleuther circle. There is microsoft paper on retnet: https://arxiv.org/abs/2307.08621 https://arxiv.org/abs/2307.08621 And the stanford heyena paper: https://arxiv.org/abs/2302.10866 https://arxiv.org/abs/2302.10866 Do note that both cited are competing alternatives for linear transformers, and in some sense competes with RWKV, in this upcoming "linear transfromer architecture" trend. --- While all the above validates architecture. It's important to separate out architecture design from dataset. The quality of an AI model answers, is equal parts as much about the dataset, than the architecture used. This is why it makes less sense comparing LLaMA2 directly with retnet/RWKV for example, due to the large differences in dataset size. > Disclaimer: I am the same Eugene in the podcast

pico_creator 3y ago

Btw, I also realise on rereading, both statements can be true at the same time. In the podcast as well, I was being very transparent that I am happily using transformers (Salesforce codegen) in production for my primary work use cases, and I am still happily doing so. My work into RWKV is to support a new usecase which transformer could not support even today. And in no means should it threaten your happiness factor in using existing AI models. I suspect in the long run, AI architecture wars may slowly mirror that of the database wars, in software development (SQL vs NoSQL vs ??) Each of them will have substantial benefits in one use case over another. And may have many overlaps, with lots of happy users for all of them =)