I'm lazy, so this isn't exhaustive. The first benchmark in the RWKV README is LAMBADA, where it got about 39% accuracy. SOTA is 90%, so that's about 6x more err

I'm lazy, so this isn't exhaustive. The first benchmark in the RWKV README is LAMBADA, where it got about 39% accuracy. SOTA is 90%, so that's about 6x more errors than SOTA. 4.5x the error rate of the first versions of GPT-3 in 2020. I didn't find LAMBADA benchmarks for GPT3.5 or GPT4, but from the GPT4 technical report, it generally makes about 2-3x fewer mistakes than GPT3.5 or prior SOTA on various benchmarks, so the difference is even more extreme. It's safe to say RWKV is not close to SOTA. Meanwhile GPT4 still really sucks at a lot of things. So by the time open source commodity hardware models do reach GPT4 level, the SOTA will be that much further ahead. The tech will be ubiquitous some day, but it's not right around the corner.