3 ms·
This is not a benchmark. They just want to give people the opportunity to try their hand at solving novel questions with AI and see what happens. If an AI compa
by yorwba 9mo ago
This is not a benchmark. They just want to give people the opportunity to try their hand at solving novel questions with AI and see what happens. If an AI company pulls a solution out of their hat that cannot be replicated with the products they make available to ordinary people, that's hardly worth bragging about and in any case it's not the point of the exercise.
- cocoto 9mo agoThey could solve the problems and train the next models with the answers, as such the future models could “solve” theses.
- fph 9mo agoThe authors mention that before publications they tested these questions on Gemini and GPT, so they have been available to the two biggest players already; they have a head start.
- data_maan 9mo agoLooks like very sloppy research.
- pickleRick243 9mo agoI don't think it's that serious...it's an interesting experiment that assumes people will take it in good faith. The idea is also of course to attach the transcript log and how you prompted the LLM so that anyone can attempt to reproduce if they wish.
- data_maan 9mo agoIf you want to do this rigorously, you should run it as a competition like the guys at the AI-MO Prize are doing on Kaggle. That way you get all the necessary data. I still think this is bro science.
- yorwba 9mo agoIf this were a competition, some people would try hard to win it. But the goal here is exploration, not exploitation. Once the answers are revealed, it's unlikely a winner will be identified, but a bunch of mathematicians who tried prompting AI with the questions might learn something from the exercise.
- data_maan 9mo agoBut everything has been explored in other datasets already. If only a bunch of mathematicians learn something, why are so many people talking about this, why is the NY Times posting about this? This is the attention economy at its worst.
- YeGoblynQueenne 9mo agoHey, sorry, totally out of context but I've always wanted to ask about the username. I keep reading it as "yoruba" in my mind. What does it mean, if I'm not being indiscreet?
- yorwba 9mo agoYou're not the first to have wondered: https://news.ycombinator.com/item?id=20730027 https://news.ycombinator.com/item?id=20730027
- YeGoblynQueenne 9mo agoWell, now that I read that comment I remembered having read it before. My mind is going.