christianstump

3 ms·

1.
by christianstump 4mo ago
I am very happy to provide any information I missed to include -- but exactly what you ask for is written in the paper's acknowledgements (no sponsorship, but some model runs).
2.
by christianstump 4mo ago
Infinitly far away. These questions are about "have you understood and can you apply existing research" not about "create new research". For humans, these two correlate quite strongly. So we ask PhD students to work on t
3.
by christianstump 4mo ago
You see yourself that your own solution is purely of theoretical nature and not at all what you wrote before, right? (And no, I am not commenting on your answer.)
4.
by christianstump 4mo ago
Please indicate which other models you would like to see included. (And I agree that the context window limitations were not reasonable to have.) Finally: running this few prompts would have been $10-20k if I would have run them myself via
5.
by christianstump 4mo ago
Well, you are correct that one should distinguish the two. But we give no indication that the questions are hard because of computational tasks and we give many indications that the problems are of theorecical nature and hard for theoretica
6.
by christianstump 4mo ago
I cannot keep answering everyone's comments of the type "Why did you consider / not consider?" or "Here are much better ideas". I promise you that we have thought quite a bit about the setup and have discussed
7.
by christianstump 4mo ago
When you write "there's a notable difference between performing a literature search versus solving something de novo", you suggest that the questions we provided can be solved doing a literature search. This is incorrect. Wha
8.
by christianstump 4mo ago
I don't understand why you expect that an answer known to the researcher but which has never been published should be in the training data. You possibly missunderstand what these problems look like -- we made them all publicly availabl
9.
by christianstump 4mo ago
You are 100% correct with your assessment of the situation. But I do not agree with either of your conclusions: 1. These questions cannot and must not be compared as being similar to homework questions. These are different leagues and possi
10.
by christianstump 4mo ago
No, they only provided large-scale model runs for us (this is explained in the ackonowledgements). These runs would have been too expensive to perform myself, so I am happy they offered to provide them.
11.
by christianstump 4mo ago
I think you are underestimating the complexity of such problems. A PhD in the exact field of research would need days to weeks to understand what the problem means and how to solve it. This is far beyond "throwing standard techniques&q
12.
by christianstump 4mo ago
Let me also add: there is zero chance of the problems being included in the training data. The results are quite impressive: leading experts struggled to write questions with well-defined unique answers on existing research that the models
13.
by christianstump 4mo ago
the difference between gpt and gemini concerning the "retry until..." can almost be ignored. I did rerun gpt a few times, but still way below what gemini was not able to answer at all.
14.
by christianstump 4mo ago
But it still remains far away from mathematics research. Solving any of the problems would not result in a new research paper.
15.
by christianstump 4mo ago
I am the leader of the study and the author of the benchmark paper: let me add: the problems are much harder than any exam question in any exam. Think of it as: a PhD student studying exactly this area of mathematics would need days to week