4 ms·
From the Nature article: > Note that the performance of GPT-4 performance on IMO problems can also be contaminated by public solutions in its training data. D
by 1024core 3y ago
From the Nature article:
> Note that the performance of GPT-4 performance on IMO problems can also be contaminated by public solutions in its training data.
Doesn't anybody proofread in Nature?
- chefandy 3y agoNot a subject matter expert, so forgive me if the question is unintentionally obtuse, but It seems like a reasonable statement. They seem to be inferring that problems with existing public solutions wouldn't be a good indicator of performance in solving novel problems-- likely a more important evaluative measure than how fast it can replicate an answer it's already seen. Since you couldn't know if that solution was in its training data, you couldn't know if you were doing that or doing a more organic series of problem solving steps, therefore contaminating it. What's the problem with saying that?
- mlyle 3y agoThey're referring to "performance of performance". Not that it's a big deal. I notice problems like this slip into my writing more and more without detection as I get older :/ I don't see the sentence in the Nature paper, though.
- chefandy 3y agoAh-- I've got super bad ADHD so I usually don't even catch things like that. I'm a terrible editor. I wouldn't be surprised if the paper authors were paying attention to this thread, and gave a call to Nature after seeing the parent comment.
- 1024core 3y agoThe text is still there. Paper: https://www.nature.com/articles/s41586-023-06747-5 https://www.nature.com/articles/s41586-023-06747-5 Section title: Geometry theorem prover baselines Second paragraph under that. 3rd sentence from the end of the para.
- 1024core 3y agoPaper: https://www.nature.com/articles/s41586-023-06747-5 https://www.nature.com/articles/s41586-023-06747-5 Section title: Geometry theorem prover baselines Second paragraph under that. 3rd sentence from the end of the para.
- dr_kiszonka 3y agoWeird. I can't find the sentence you are quoting in the paper.
- 1024core 3y agoPaper: https://www.nature.com/articles/s41586-023-06747-5 https://www.nature.com/articles/s41586-023-06747-5 Section title: Geometry theorem prover baselines Second paragraph under that. 3rd sentence from the end of the para.
- dr_kiszonka 3y agoThank you.