5 ms·

This is certainly impressive, but whenever IMO is brought up, a caveat should be put out: medals are awarded to 50% of the participants (high school students),

This is certainly impressive, but whenever IMO is brought up, a caveat should be put out: medals are awarded to 50% of the participants (high school students), with 1:2:3 ratio between gold, silver and bronze. That puts all gold and silver medalists among the top 25% of the participants. That means that "AI solves IMO problems better than 75% of the students", which is probably even more impressive. But, "minutes for one problem and up to 3 days for each remaining problem" means that this is unfortunately not a true representation either. If these students were given up to 15 days (5 problems at "up to 3 days each") instead of 9h, there would probably be more of them that match or beat this score too. It really sounds like AI solved only a single problem in the 9h students get, so it certainly would not be even close to the medals. What's the need to taint the impressive result with apples-to-oranges comparison? Why not be more objective and report that it took longer but was able to solve X% of problems (or scored X out of N points)?

muglug 2y ago
> What's the need to taint the impressive result with apples-to-oranges comparison? Most of DeepMind’s research is a cost-centre for the company. These press releases help justify the continued investment both to investors and to the wider public.
- utopcell 2y ago
  > Most of DeepMind’s research is a cost-centre for the company. The effect of establishing oneself as the thought leader in a field is enormous. For example, IBM's stock went up 15% the month after they beat Kasparov.
- mensetmanusman 2y ago
  Cost centers are profit centers when R&D is successful.
Davidzheng 2y ago
In my opinion (not Google s) the only reason they didn't get gold this year (apart from being unlucky on problem selection) is that they didn't want to try for any partial credit in P3 and P5. They are so close to the cut off and usually contestants with a little bit of progress can get 1 point. But i guess they didn't want to get a gold on a technicality--it would be bad press. So they settled in a indisputable silver
- dooglius 2y ago
  I don't believe anything was graded by the IMO, Google is just giving itself 7 for anything proved in Lean (which is reasonable IMO), so they can't really try for partial credit so much as choose not to report a higher self-graded number.
  - utopcell 2y ago
    From the article: "Our solutions were scored according to the IMO’s point-awarding rules by prominent mathematicians Prof Sir Timothy Gowers, an IMO gold medalist and Fields Medal winner, and Dr Joseph Myers, a two-time IMO gold medalist and Chair of the IMO 2024 Problem Selection Committee."
    - dooglius 2y ago
      Fair enough. Although I think the question is whether P3/P5 were given zero points vs not evaluated vs evaluated but not published. I don't think it is surprising that Lean-verified proofs get a 7.
      - utopcell 2y ago
        It is an interesting question: how this new system quantifies progress and whether it can detect that a chain of reasoning is close to the solution. Although it was just 2 points shy of a golden medal (1 point partial credit per problem not solved?), I'd suspect that the team would want a clean victory, given the magnitude of the claim. I bet next year's results would be 42/42. Deep Mind is unstoppable!
- llwu 2y ago
  Partial credit is quite challenging to earn, per: https://www.imo-official.org/year_statistics.aspx?year=2024 https://www.imo-official.org/year_statistics.aspx?year=2024
  - Davidzheng 2y ago
tardygrade 2y ago
One key difference between giving humans more time and giving computer programs more time is that historically we have had more success making the latter run faster than the former.
acchow 2y ago
But computers get faster each year, so even with zero progress in actual AI, this will reach human-student speeds in a few years (need a 40x speed up)
- awesomeMilou 2y ago
  Could you explain where the 40x speedup comes from, given that literally the biggest problem in semi conductors right now is smaller node size?
  - cma 2y ago
    Just bigger chip area at lower clock could do it too if there is no threshold on cost anyway and they were limited by power. Google would hit enough accumulated production of AI chip area and cluster build out even if Moore's law stopped. There will likely be big algorithmic improvments too.
  - lupire 2y ago
    Improved parallelization. More cost in less time. This is Nvidia's main area of research now.
    - knotthebest 2y ago
      Still, very marginal improvements. There is also Amdahl’s Law which limits the speed gains of parallelisation.
NiloCK 2y ago
> medals are awarded to 50% of the participants (high school students) In case this confuses anyone: the high school students in question are not a standard sample of high school students. AFAIK, they are teams of the ~6 strongest competitive problem solving high school students from each country.
SonOfLilit 2y ago
I have met IMO competitors. They are insanely smart. I wouldn't imagine it's possible to be that smart before I started hanging in these circles. So more like 25% of 0.01% of high school students. Time is not a very interesting dimension here, because humans don't use the same CPU as huge GPU clusters. The binary "is it able to reach a solution given enough resources?" is more interesting (for GPT/Claude the answer is a clear negative).
- lupire 2y ago
  But many more of those students (including many not in the contest) could solve those problems given more time than the 4.5hr sessions. If these problems were important to solve, redundantly by thousands or millions of people (like the real work that most people do), far more people would put in the effort to learn how to solve these problems. It's just a weird comparison. Contests are very artificially structured in ways that don't make sense for comparing to computers.
  - badrunaway 2y ago
    I think what we need to look at is that AI systems were not able to do it before and now they are able to do it. Sooner these systems with millions of dollars of compute will just scale the algorithms and beat humans at this as well - just like they did for chess, go, shogi etc.
- astromaniak 2y ago
  I've seen chess players, they are smart too. So what? Specialized model on commodity hardware beats most of them, if not all. This doesn't mean model is smarter. The same could be here. Is it possible that solutions are one step from model's database, or training set. Humans don't remember this much and have to do many more actions in their heads. In other words model which can solve any/most school / college exam problem isn't necessarily smart. It can be just a database, or, in fact, a lookup table. Smarter version can be a lookup + 1 step test. Not saying it's bad, but it doesn't scale to less formalized domains. BTW, in this case formalization was done by humans.
- necovek 2y ago
xkcd1963 2y ago
Because that wouldn't be hype enough.
asah 2y ago
The complexity of truth doesn't fit in a headline, let alone attract enough clicks to win the competition for eyeballs and virality - which means that even if someone somehow succeeded in telling the whole truth in a headline, that article would lose in the race for eyeballs to the clickbait-headline version, which means your eyeballs would (statistically) never see it, and only the clickbait version. I'd say "welcome to the web" but this was true in 1800s newspapers as well.

muglug 2y ago

> What's the need to taint the impressive result with apples-to-oranges comparison? Most of DeepMind’s research is a cost-centre for the company. These press releases help justify the continued investment both to investors and to the wider public.

utopcell 2y ago
> Most of DeepMind’s research is a cost-centre for the company. The effect of establishing oneself as the thought leader in a field is enormous. For example, IBM's stock went up 15% the month after they beat Kasparov.
mensetmanusman 2y ago
Cost centers are profit centers when R&D is successful.

Davidzheng 2y ago

In my opinion (not Google s) the only reason they didn't get gold this year (apart from being unlucky on problem selection) is that they didn't want to try for any partial credit in P3 and P5. They are so close to the cut off and usually contestants with a little bit of progress can get 1 point. But i guess they didn't want to get a gold on a technicality--it would be bad press. So they settled in a indisputable silver

dooglius 2y ago
I don't believe anything was graded by the IMO, Google is just giving itself 7 for anything proved in Lean (which is reasonable IMO), so they can't really try for partial credit so much as choose not to report a higher self-graded number.
- utopcell 2y ago
  From the article: "Our solutions were scored according to the IMO’s point-awarding rules by prominent mathematicians Prof Sir Timothy Gowers, an IMO gold medalist and Fields Medal winner, and Dr Joseph Myers, a two-time IMO gold medalist and Chair of the IMO 2024 Problem Selection Committee."
  - dooglius 2y ago
    Fair enough. Although I think the question is whether P3/P5 were given zero points vs not evaluated vs evaluated but not published. I don't think it is surprising that Lean-verified proofs get a 7.
    - utopcell 2y ago
      It is an interesting question: how this new system quantifies progress and whether it can detect that a chain of reasoning is close to the solution. Although it was just 2 points shy of a golden medal (1 point partial credit per problem not solved?), I'd suspect that the team would want a clean victory, given the magnitude of the claim. I bet next year's results would be 42/42. Deep Mind is unstoppable!
llwu 2y ago
Partial credit is quite challenging to earn, per: https://www.imo-official.org/year_statistics.aspx?year=2024 https://www.imo-official.org/year_statistics.aspx?year=2024
- Davidzheng 2y ago

tardygrade 2y ago

One key difference between giving humans more time and giving computer programs more time is that historically we have had more success making the latter run faster than the former.

acchow 2y ago

But computers get faster each year, so even with zero progress in actual AI, this will reach human-student speeds in a few years (need a 40x speed up)

awesomeMilou 2y ago
Could you explain where the 40x speedup comes from, given that literally the biggest problem in semi conductors right now is smaller node size?
- cma 2y ago
  Just bigger chip area at lower clock could do it too if there is no threshold on cost anyway and they were limited by power. Google would hit enough accumulated production of AI chip area and cluster build out even if Moore's law stopped. There will likely be big algorithmic improvments too.
- lupire 2y ago
  Improved parallelization. More cost in less time. This is Nvidia's main area of research now.
  - knotthebest 2y ago
    Still, very marginal improvements. There is also Amdahl’s Law which limits the speed gains of parallelisation.

NiloCK 2y ago

> medals are awarded to 50% of the participants (high school students) In case this confuses anyone: the high school students in question are not a standard sample of high school students. AFAIK, they are teams of the ~6 strongest competitive problem solving high school students from each country.

SonOfLilit 2y ago

I have met IMO competitors. They are insanely smart. I wouldn't imagine it's possible to be that smart before I started hanging in these circles. So more like 25% of 0.01% of high school students. Time is not a very interesting dimension here, because humans don't use the same CPU as huge GPU clusters. The binary "is it able to reach a solution given enough resources?" is more interesting (for GPT/Claude the answer is a clear negative).

lupire 2y ago
But many more of those students (including many not in the contest) could solve those problems given more time than the 4.5hr sessions. If these problems were important to solve, redundantly by thousands or millions of people (like the real work that most people do), far more people would put in the effort to learn how to solve these problems. It's just a weird comparison. Contests are very artificially structured in ways that don't make sense for comparing to computers.
- badrunaway 2y ago
  I think what we need to look at is that AI systems were not able to do it before and now they are able to do it. Sooner these systems with millions of dollars of compute will just scale the algorithms and beat humans at this as well - just like they did for chess, go, shogi etc.
astromaniak 2y ago
I've seen chess players, they are smart too. So what? Specialized model on commodity hardware beats most of them, if not all. This doesn't mean model is smarter. The same could be here. Is it possible that solutions are one step from model's database, or training set. Humans don't remember this much and have to do many more actions in their heads. In other words model which can solve any/most school / college exam problem isn't necessarily smart. It can be just a database, or, in fact, a lookup table. Smarter version can be a lookup + 1 step test. Not saying it's bad, but it doesn't scale to less formalized domains. BTW, in this case formalization was done by humans.
necovek 2y ago

xkcd1963 2y ago

Because that wouldn't be hype enough.

asah 2y ago

The complexity of truth doesn't fit in a headline, let alone attract enough clicks to win the competition for eyeballs and virality - which means that even if someone somehow succeeded in telling the whole truth in a headline, that article would lose in the race for eyeballs to the clickbait-headline version, which means your eyeballs would (statistically) never see it, and only the clickbait version. I'd say "welcome to the web" but this was true in 1800s newspapers as well.