4 ms·

#4 (private test set) doesn't get used for any public model testing. It is only used on the Kaggle leaderboard where no internet access is allowed.

zamadatix 2y ago
Sorry, I probably phrased the question poorly. My question is more along the lines of "when you already scored e.g. OpenAI's o3 on ARC AGI 2 how did you guarantee OpenAI can't just look at its server logs to see question set 4"?
- gkamradt 2y ago
  Ah yes, two things 1. We had a no-data retention agreement with them. We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing 2. We only tested o3 against the semi-private set. We didn't test it with the private eval.
  - zamadatix 2y ago
    Makes sense, particularly part 2 until "the final results" are needed. Thanks for taking the time to answer my question!
  - YeGoblynQueenne 2y ago
    >> We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing Yuri Geller assured us he was bending the spoons with his mind. Somehow it was only when the Amazing Randi was present that Yuri Geller couldn't bend the spoons with his mind.
    - levocardia 2y ago
      Ironically "I have a magic AI test but nobody is allowed to use it" is a lot closer to the Yuri Geller situation. Tests are meant to be taken, that should be clear. And...maybe this does not apply in the academic domain, but to some extent if you cheat on an AI test "you're only cheating yourself."
      - Jensson 2y ago
        > but to some extent if you cheat on an AI test "you're only cheating yourself." You cheat investors.
        anshumankmr 2y ago
        And end users and developers and the general public too... But here is the thing, I feel that even if its rote memorizing why GPT4o couldn't perform just as well on ArcAGI 1 on it or did the "reasoning" help in any way?
  - QuadmasterXLII 2y ago
    Are you aware that OpenAI brazenly lied and went back on its word about its corporate structure, board governance, and for-profit status, and of the opinion that your data sharing agreement is different and less likely to be ignored? Or are you at step zero where you aren’t considering malfeasance as a possibility at all?

zamadatix 2y ago

Sorry, I probably phrased the question poorly. My question is more along the lines of "when you already scored e.g. OpenAI's o3 on ARC AGI 2 how did you guarantee OpenAI can't just look at its server logs to see question set 4"?

gkamradt 2y ago
Ah yes, two things 1. We had a no-data retention agreement with them. We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing 2. We only tested o3 against the semi-private set. We didn't test it with the private eval.
- zamadatix 2y ago
  Makes sense, particularly part 2 until "the final results" are needed. Thanks for taking the time to answer my question!
- YeGoblynQueenne 2y ago
  >> We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing Yuri Geller assured us he was bending the spoons with his mind. Somehow it was only when the Amazing Randi was present that Yuri Geller couldn't bend the spoons with his mind.
  - levocardia 2y ago
    Ironically "I have a magic AI test but nobody is allowed to use it" is a lot closer to the Yuri Geller situation. Tests are meant to be taken, that should be clear. And...maybe this does not apply in the academic domain, but to some extent if you cheat on an AI test "you're only cheating yourself."
    - Jensson 2y ago
      > but to some extent if you cheat on an AI test "you're only cheating yourself." You cheat investors.
      - anshumankmr 2y ago
        And end users and developers and the general public too... But here is the thing, I feel that even if its rote memorizing why GPT4o couldn't perform just as well on ArcAGI 1 on it or did the "reasoning" help in any way?
- QuadmasterXLII 2y ago
  Are you aware that OpenAI brazenly lied and went back on its word about its corporate structure, board governance, and for-profit status, and of the opinion that your data sharing agreement is different and less likely to be ignored? Or are you at step zero where you aren’t considering malfeasance as a possibility at all?