3 ms·

That project isn’t close to the SoTA models. It performs well, but not close to GPT-4 for example. “Almost the same performance as gpt” always ends up meaning i

That project isn’t close to the SoTA models. It performs well, but not close to GPT-4 for example. “Almost the same performance as gpt” always ends up meaning it performs like one of the smaller GPT models.

nico 4y ago
Sure but it just came out. And at the rate things are going, it will likely be just as good as GPT-3 or 4 soon. Also what is “close”? The benchmarks I saw showed around 80% accuracy compared to GPT.
- Aperocky 4y ago
  > it will likely be just as good as GPT-3 or 4 soon. It just came out, but in 4 months of development this vehicle should be able to do 2000 miles per gallon instead of 20 miles per gallon. GPT is not magic, it's performance is linear with its size.
  - nico 4y ago
    It is not about GPT, it’s a whole revolution that is happening. New models are coming out almost every day. Including some that have almost same performance as GPT but are 100x faster to train and are smaller (look at this:https://news.ycombinator.com/item?id=35281026 https://news.ycombinator.com/item?id=35281026). You can believe or deny whatever you want. The technology is moving way faster than we can keep up with and the speed is accelerating.
- 6gvONxR4sf7o 4y ago
  I'm lazy, so this isn't exhaustive. The first benchmark in the RWKV README is LAMBADA, where it got about 39% accuracy. SOTA is 90%, so that's about 6x more errors than SOTA. 4.5x the error rate of the first versions of GPT-3 in 2020. I didn't find LAMBADA benchmarks for GPT3.5 or GPT4, but from the GPT4 technical report, it generally makes about 2-3x fewer mistakes than GPT3.5 or prior SOTA on various benchmarks, so the difference is even more extreme. It's safe to say RWKV is not close to SOTA. Meanwhile GPT4 still really sucks at a lot of things. So by the time open source commodity hardware models do reach GPT4 level, the SOTA will be that much further ahead. The tech will be ubiquitous some day, but it's not right around the corner.

nico 4y ago

Sure but it just came out. And at the rate things are going, it will likely be just as good as GPT-3 or 4 soon. Also what is “close”? The benchmarks I saw showed around 80% accuracy compared to GPT.

Aperocky 4y ago
> it will likely be just as good as GPT-3 or 4 soon. It just came out, but in 4 months of development this vehicle should be able to do 2000 miles per gallon instead of 20 miles per gallon. GPT is not magic, it's performance is linear with its size.
- nico 4y ago
  It is not about GPT, it’s a whole revolution that is happening. New models are coming out almost every day. Including some that have almost same performance as GPT but are 100x faster to train and are smaller (look at this:https://news.ycombinator.com/item?id=35281026 https://news.ycombinator.com/item?id=35281026). You can believe or deny whatever you want. The technology is moving way faster than we can keep up with and the speed is accelerating.
6gvONxR4sf7o 4y ago
I'm lazy, so this isn't exhaustive. The first benchmark in the RWKV README is LAMBADA, where it got about 39% accuracy. SOTA is 90%, so that's about 6x more errors than SOTA. 4.5x the error rate of the first versions of GPT-3 in 2020. I didn't find LAMBADA benchmarks for GPT3.5 or GPT4, but from the GPT4 technical report, it generally makes about 2-3x fewer mistakes than GPT3.5 or prior SOTA on various benchmarks, so the difference is even more extreme. It's safe to say RWKV is not close to SOTA. Meanwhile GPT4 still really sucks at a lot of things. So by the time open source commodity hardware models do reach GPT4 level, the SOTA will be that much further ahead. The tech will be ubiquitous some day, but it's not right around the corner.