3 ms·
There is something of a middle ground here for benchmark skepticism. Big companies wouldn't want a massive divergence between benchmarks and real performance th
by christianqchung 1y ago
There is something of a middle ground here for benchmark skepticism. Big companies wouldn't want a massive divergence between benchmarks and real performance that people could actually notice, and I'd argue for the most part that this hasn't happened too much (although above I posted a problem with Qwen and ARC). However, finetunes by random people/groups don't carry the same downside so I'm basically skeptical of all finetunes before using them for a particular case.
- energy123 1y agoI don't believe these companies see their customers as being able to tell the difference between a real GPQA score and a GPQA score that's fudged upwards by 10%. Look at Elon Musk presenting himself to the world as a Path of Exile expert when in reality he likely hired an expert to level up his account while he himself is an amateur. They think we are idiots and will lie to us to capture market share and lock us into their ecosystem.
- christianqchung 1y agoThat's true, I certainly wouldn't be able to tell. I was thinking on the order of a 20% score vs 70%, but I realize that's not a very compelling range for my point when people are boasting about <5% shifts.