4 ms·
My experience reflects this too. My hunch is that GPT-4o was trained to game the benchmarks rather than output higher quality content. In theory the benchmarks
by wfme 2y ago
My experience reflects this too. My hunch is that GPT-4o was trained to game the benchmarks rather than output higher quality content.
In theory the benchmarks should be a pretty close proxy for quality, but that doesn't match my experience at all.
- margorczynski 2y agoA problem with a lot of benchmarks is that they are out in the open so the model basically trains to game them instead of actually acquiring knowledge that would let it solve it. Probably private benchmarks that are not in the training set of these models should give better estimates about their general performance.