3 ms·
And like with most other competition/benchmark, the result is likely optimizing for the benchmark and not the wider goal ;-). It’s difficult to get a serious ef
by woodson 2y ago
And like with most other competition/benchmark, the result is likely optimizing for the benchmark and not the wider goal ;-). It’s difficult to get a serious effort without people trying to game the benchmark.
- luke-stanley 2y agoPeople could try but it would not help with the withheld datasets so much, and it would be possible to add more to it. If the withheld data was closed, and only available to an assessment system, gaming that would be pretty difficult. Scale.com's SEAL Leaderboards take a similar approach. The ARC Prize still exists too, and it's waiting for winners. See https://scale.com/leaderboard https://scale.com/leaderboard