3 ms·
> All the top 5 players on the final leaderboard [...] used LLMs for most of their solutions. Note that the leaderboard points are given out on the time taken
by michaelt 2y ago
> All the top 5 players on the final leaderboard [...] used LLMs for most of their solutions.
Note that the leaderboard points are given out on the time taken to produce a correct answer. 100 points for the first to submit an answer, 99 for the second, 98 for third and so on. No points if you're not in the first 100.
So if an LLM fails on 5 problems, but for the other 20 it can take a 600-word problem statement and solve it in 12 seconds? It'll rack up loads of points.
Whereas the best human programmer you know might be able to solve all 25 problems taking around 15 minutes per problem. On most days they would have zero points, as all 100 points finishes go to LLMs.