3 ms·
Surely this could be better pitched to researchers as just another AI benchmark, a bit like ARC Prize? ;) There could be some exiting C projects that are alread
by luke-stanley 2y ago
Surely this could be better pitched to researchers as just another AI benchmark, a bit like ARC Prize? ;)
There could be some exiting C projects that are already public, with tests for feedback during development iteration and some holdout tests, and some holdout projects too with a leaderboard and prizes.
For preferences about converted code quality, both automated assesment and human preferences could be ranked with Elo? Kaggle is made for this sort of thing I think?
I'm sure Google Deepmind and others have some MCTS agents that could do a great job with a bit of effort.
- woodson 2y agoAnd like with most other competition/benchmark, the result is likely optimizing for the benchmark and not the wider goal ;-). It’s difficult to get a serious effort without people trying to game the benchmark.
- luke-stanley 2y agoPeople could try but it would not help with the withheld datasets so much, and it would be possible to add more to it. If the withheld data was closed, and only available to an assessment system, gaming that would be pretty difficult. Scale.com's SEAL Leaderboards take a similar approach. The ARC Prize still exists too, and it's waiting for winners. See https://scale.com/leaderboard https://scale.com/leaderboard