3 ms·
Related: https://news.ycombinator.com/item?id=44613840 https://news.ycombinator.com/item?id=44613840
by blendergeek 1y ago
Related: https://news.ycombinator.com/item?id=44613840 https://news.ycombinator.com/item?id=44613840
- untitled2 1y agoExactly. Whom to believe?
- JohnKemeny 1y agoThe last time someone claimed a medal in an olympiad like this, turned out they manually translated the problem into Lean and then ran a brute force search algorithm to find a proof. For 60 hours. On a supercomputer. Meanwhile high schoolers get a piece of paper and 4.5 hours.
- wslh 1y agoEven though chess is now effectively solved against human players, I still remember Kasparov's suspicion that one of Deep Blue's moves had a human touch. It was never proven or disproven, but I trust Kasparov's deep intuition amplified by Kasparov requesting access to Deep Blue’s logs, and IBM refusing to share them in full. For more discussions see [1][2][3]. [1] https://chess.stackexchange.com/questions/9959/did-deep-blue-play-like-a-human https://chess.stackexchange.com/questions/9959/did-deep-blue... [2] https://nautil.us/why-the-chess-computer-deep-blue-played-like-a-human-235130/ https://nautil.us/why-the-chess-computer-deep-blue-played-li... [3] https://en.chessbase.com/post/deep-blue-s-cheating-move https://en.chessbase.com/post/deep-blue-s-cheating-move
- throwawaymaths 1y agokinda wild that an llm cant translate to lean?
- changoplatanero 1y agoBoth are true. One spent $400 in compute and the other one spent a lot more.
- masterjack 1y agoExactly. And presumably had a more sophisticated harness around the model, longer reasoning chains, best of N, self judging, etc
- kenjackson 1y agoOpenAI achieved Gold on an unreleased model. GPT-5. Read the tweets and they explain what they did.
- idiotsecant 1y agoActually, I did it a year ago but I just don't want to release my model.
- e1g 1y agoOpenAI explicitly said it’s not GPT-5 but another experimental research model https://x.com/alexwei_/status/1946477756738629827?s=46 https://x.com/alexwei_/status/1946477756738629827?s=46
- kenjackson 1y agoThanks. I parsed that wrong. In either case not the same thing Math Arena used.
- raincole 1y agoNote that it's two different things: This OP claims the publicly available models all failed to get Bronze. OpenAI tweet claims there is an unreleased model that can get Gold.
- dmitrygr 1y agoMy (unreleased) cat did even better than the OpenAI model. No you cannot see. Yes you have to trust me. Now gimme more money.
- raincole 1y agoI don't know the details (of course, it's unreleased), but note that MathArena evaluated "average of 4 attempts", and limited token usages to 64k. OpenAI likely had unlimited tokens, and evaluated "best of N attempts."
- amelius 1y agoThat's a claim that is far less plausible. OpenAI could have thrown more resources at the problem and I would be surprised if that didn't improve the results.
- klabb3 1y agoWow, that’s incredible. Cats are progressing so fast, especially unreleased cats seem to be doing much better. My two orange kitties aren’t doing well on math problems but obviously that’s because I’m not prompting the right way – any day now. If I ever get it to work, I’ll be sure to share the achievements on X, while carefully avoiding explaining how I did it or provide any data that can corroborate the claims.
- sigmoid10 1y agoI'd also be highly wary of the method they used because of statements like this: >we note that the vast majority of its answers simply stated the final answer without additional justification While the reasoning steps are obviously important for judging human participant answers, none of the current big-game providers disclose their actual reasoning tokens. So unless they got direct internal access to these models from the big companies (which seems highly unlikely), this might be yet another failed study designed to (of which we have seen several in recent months, even by serious parties).