3 ms·

I think is feasible. The description even says is designed to save on vram[1]. I don't get the other comments about needing more vram than a 3090. Also, Neural

I think is feasible. The description even says is designed to save on vram[1]. I don't get the other comments about needing more vram than a 3090. Also, Neuralmagic may run their sparsification on ARM cpu's in the future, so keep an eye. 1. ChatRWKV v2: with "stream" and "split" strategies. 3G VRAM is enough to run RWKV 14B :)

IanCal 4y ago
You have to split it up which slows it down a lot. The 14B model doesn't fit fully on a 3090, though the 7B fits easily and is very fast. Other replies either may have meant this or thought the original comment was about llama.