3 ms·
This is the best side-by-side comparison I've seen so far, we're still waiting for it to become available for download, then we should get much better analyses.
by verdverm 3mo ago
This is the best side-by-side comparison I've seen so far, we're still waiting for it to become available for download, then we should get much better analyses.
https://news.ycombinator.com/item?id=48999291 https://news.ycombinator.com/item?id=48999291
I do agree that the benchmarks do not tell much of the story, but this applies to the closed models as well ime.
- braebo 3mo agoI’m not kidding when I say bullshit bench and simple bench are the only benchmarks that reflect real world utility in a way that matches my hundreds of hours of experience with frontier models: https://petergpt.github.io/bullshit-benchmark/viewer/index.v2.html https://petergpt.github.io/bullshit-benchmark/viewer/index.v... That said, while I find it hard to trust fireworks given their conflict of interest, their article is pretty good.
- verdverm 3mo agoThat is an interesting benchmark! Though I wonder if it is probing at certain behaviors, which may be more revealing to probe at? Will keep it handy, thanks for the share! I felt better about Fireworks leadership after hearing some of them more candidly on podcasts (there are 7 founders!)