3 ms·
>I feel the opposite, and pretty much every metric we have shows basically linear improvement of these models over time. Wait, what kind of metric are you talk
by attemptone 1y ago
>I feel the opposite, and pretty much every metric we have shows basically linear improvement of these models over time.
Wait, what kind of metric are you talking about? When I did my masters in 2023 SOTA models where trying to push the boundaries by minuscule amounts. And sometimes blatantly changing the way they measure "success" to beat the previous SOTA
- mountainriver 1y agoAlmost every single major benchmark, and yes progress is incremental but it adds up, this has always been the case
- attemptone 1y agoWe were talking about linear improvements and I have yet to see it
- mountainriver 1y agocheck the benchmarks or make one of your own
- attemptone 1y agoI checked the BlEU-Score and Perplexity of popular models and both have stagnated around 2021. As a disclaimer this was a cursory check and I didn't dive into the details of how individuals scores were evaluated.
- mountainriver 1y agoon what benchmarks? pretty much every major one is linear improvement