3 ms·

Why would they exclude the solution to the problem and give themselves a handicap that the AI does not have?

by devmor 1y ago

scotty79 1y ago
Because AI didn't have solutions for this year's problem in its training materials. Same way that students participating in this year's Math Olympiad didn't. Guys, don't you know anything about how such competitions work? You get limited time and a set of problems to solve. New problems, that weren't available anywhere before they were given to you. That might be the issue. How can you appreciate what the AI achieved if you don't know anything about it beyond the name?
- devmor 1y ago
  Per Gregor Dolinar, President of the IMO: > “It is very exciting to see progress in the mathematical capabilities of AI models, but we would like to be clear that the IMO cannot validate the methods, including the amount of compute used or whether there was any human involvement, or whether the results can be reproduced. What we can say is that correct mathematical proofs, whether produced by the brightest students or AI models, are valid.”
  - fragmede 1y ago
    Which is to say, they didn't give the AI model the answer, the AI model produced one without having seen it, and that it's valid. It might have taken extra time or had outside help, or otherwise cheated, but it didn't have the answer given to it to merely reproduce.
    - devmor 1y ago
      No such stipulation or claim is made. The only verified claim is that the proof is valid.
      - fragmede 1y ago
        but also no stipulation otherwise is made by the contest runners. As the runners of a contest, I would presume they didn't give the answer to any of the test takers before they took the test because that would defeat the whole point of having the contest! If you think people/machines are cheating absent an explicit claim as such, I can't help you, but that seems unlikely to me. It's entirely possible that both Google and OpenAI independently bribed a question writer for the answers and fed that into the LLM to be used as training data, and have not been discovered doing so, and that only then was the LLM able to generate the correct answer, but that seems a bit far fetched to me.
  - scotty79 1y ago
    So earlier you said, AI answered with scraped answers. Since you now realize that was not the case, what's your belief now? That OpenAI (and Google) did a full on mechanical turk scam? Because that's what the fragment you cited is saying. That MO can't assure that the answers provided by AI companies were really generated by AI (because they have no way of knowing that).
    - devmor 1y ago
      > Because that's what the fragment you cited is saying. That MO can't assure that the answers provided by AI companies were really generated by AI It says nothing except that the proofs are valid and absolutely no assurance about them can be made. It does not tell you that the answers were not scraped.
      - scotty79 1y ago
        I do not need anyone's assurance that if I'm gonna step out of the window, I'm gonna fall. Creating accurate (and predictive) model of reality in your head based on incomplete data is an essential skill. Your model is bad. Either ingest more data or train more on the data you already got.

Why would they exclude the solution to the problem and give themselves a handicap that the AI does not have?

by devmor 1y ago

Why would they exclude the solution to the problem and give themselves a handicap that the AI does not have?

Because AI didn't have solutions for this year's problem in its training materials. Same way that students participating in this year's Math Olympiad didn't. Guys, don't you know anything about how such competitions work? You get limited time and a set of problems to solve. New problems, that weren't available anywhere before they were given to you. That might be the issue. How can you appreciate what the AI achieved if you don't know anything about it beyond the name?

devmor 1y ago
Per Gregor Dolinar, President of the IMO: > “It is very exciting to see progress in the mathematical capabilities of AI models, but we would like to be clear that the IMO cannot validate the methods, including the amount of compute used or whether there was any human involvement, or whether the results can be reproduced. What we can say is that correct mathematical proofs, whether produced by the brightest students or AI models, are valid.”
- fragmede 1y ago
  Which is to say, they didn't give the AI model the answer, the AI model produced one without having seen it, and that it's valid. It might have taken extra time or had outside help, or otherwise cheated, but it didn't have the answer given to it to merely reproduce.
  - devmor 1y ago
    No such stipulation or claim is made. The only verified claim is that the proof is valid.
    - fragmede 1y ago
      but also no stipulation otherwise is made by the contest runners. As the runners of a contest, I would presume they didn't give the answer to any of the test takers before they took the test because that would defeat the whole point of having the contest! If you think people/machines are cheating absent an explicit claim as such, I can't help you, but that seems unlikely to me. It's entirely possible that both Google and OpenAI independently bribed a question writer for the answers and fed that into the LLM to be used as training data, and have not been discovered doing so, and that only then was the LLM able to generate the correct answer, but that seems a bit far fetched to me.
- scotty79 1y ago
  So earlier you said, AI answered with scraped answers. Since you now realize that was not the case, what's your belief now? That OpenAI (and Google) did a full on mechanical turk scam? Because that's what the fragment you cited is saying. That MO can't assure that the answers provided by AI companies were really generated by AI (because they have no way of knowing that).
  - devmor 1y ago
    > Because that's what the fragment you cited is saying. That MO can't assure that the answers provided by AI companies were really generated by AI It says nothing except that the proofs are valid and absolutely no assurance about them can be made. It does not tell you that the answers were not scraped.
    - scotty79 1y ago
      I do not need anyone's assurance that if I'm gonna step out of the window, I'm gonna fall. Creating accurate (and predictive) model of reality in your head based on incomplete data is an essential skill. Your model is bad. Either ingest more data or train more on the data you already got.