4 ms·

I wouldn't be surprised if ≈7% of GPQA Diamond questions simply have the wrong answer in the ground truth data, so that getting such a question correct is grade

by yorwba 3mo ago

I wouldn't be surprised if ≈7% of GPQA Diamond questions simply have the wrong answer in the ground truth data, so that getting such a question correct is graded as an error. Most machine-learning benchmarks are rather badly validated.

seizethecheese 3mo ago
Yep! I found this interesting article after banging my head against a wall for a long time: https://epoch.ai/gradient-updates/gpqa-diamond-whats-left https://epoch.ai/gradient-updates/gpqa-diamond-whats-left