3 ms·
Fixed benchmarks will be debunked eventually I'd think. Better to use synthetic problems. Too easy to game the numbers, and too easy to baselessly accuse compa
by dpweb 23d ago
Fixed benchmarks will be debunked eventually I'd think. Better to use synthetic problems.
Too easy to game the numbers, and too easy to baselessly accuse companies of gaming the numbers, not to mention how you even define that.
- gunalx 23d agoWouldn't work. A dynamic but verifiable problem. Is just a perfect target for a RL environment. If you don't have the verifiable part the benchmark is useless, or really expensive with human review. (Or just open ended)
- dpweb 22d agoThere doesn't need to be a single correct response. Generate "novel" problems with a set of acceptable solutions/outcomes, then verify that the response satisfies the criteria.