4 ms·

Personally, the RNN "mode" runs substantially faster with similar quality to other models of the same size. RWKV is limited by compute and thus model size at pr

Personally, the RNN "mode" runs substantially faster with similar quality to other models of the same size. RWKV is limited by compute and thus model size at present which makes it hard to suggest how it will stack up against models like falcon and llama2 in their larger sizes. There are also trade offs around their specific attention mechanism. It's approximate and consists of an exponential decay across time steps and some constant that represents the magnitude of the attention at that time step. The decay means for long running contexts it in theory will loses some of the ability to recall nuance you see with perfect attention on longer sequences. This isn't a specific problem of RWKV, but really any decay based attention. A perk of the formulation is the attention computation is linear in time complexity and only requires one additional serial scan at the end, so you get nicer scaling properties than quadratic perfect attention for the trade off in lossy attention. The examples in the original paper don't show this to be a problem, but they also show much shorter sequences. In the real world its a toss up, I'd expect it to be faster but more lossy for longer context windows. TLDR: It has some teeth, but I think saying it's threatening is wishful thinking to some degree. Good model, looking forward to seeing larger sizes.

pico_creator 3y ago
Agreed that as a whole, we do need more public benchmarks (both transformers and RWKV), with much longer sequence length. I view this as a case of the "benchmarks" and "datasets" not keeping up with the current progress, as the vast majority of benchmarks presumes the limit of up to 2k tokens. Which was cutting edge last year, but isn't today. Internally we do have some benchmarking for how the decay over time affects lossy memory, and we are making major improvements in this direction for v5, and aim for no loss within the 8k to 100k and beyond context sizes. And hopefully with time, there will be more formalised standard benchmarks for larger context sizes. Edit/PS: I believe until we get a sponsor for the compute required to build on similar scale and dataset compared to LLaMA2, so that we can have that "showdown" - we would be stuck - in its non threatening status. As there will always be that shadow of doubt, if it can scale to the next milestone that transformers have already crossed.

pico_creator 3y ago

Agreed that as a whole, we do need more public benchmarks (both transformers and RWKV), with much longer sequence length. I view this as a case of the "benchmarks" and "datasets" not keeping up with the current progress, as the vast majority of benchmarks presumes the limit of up to 2k tokens. Which was cutting edge last year, but isn't today. Internally we do have some benchmarking for how the decay over time affects lossy memory, and we are making major improvements in this direction for v5, and aim for no loss within the 8k to 100k and beyond context sizes. And hopefully with time, there will be more formalised standard benchmarks for larger context sizes. Edit/PS: I believe until we get a sponsor for the compute required to build on similar scale and dataset compared to LLaMA2, so that we can have that "showdown" - we would be stuck - in its non threatening status. As there will always be that shadow of doubt, if it can scale to the next milestone that transformers have already crossed.