3 ms·

If you are talking about ARC benchmark, then o3-low doesn't look that special if you take into account there are plenty of finetuned models with much smaller re

If you are talking about ARC benchmark, then o3-low doesn't look that special if you take into account there are plenty of finetuned models with much smaller resources achieved 40-50% results on private set (not semi-private like o3-low).

famouswaffles 2y ago
- I'm not just talking about ARC. On frontier Math, we have 2 scores, one with pass@1 and another with consensus vote with 64 samples. Both scores are much better than previous Sota. - Also apparently, ARC wasn't a special fine-tune but rather some of the training set in the corpus for pre-training.
- riku_iki 2y ago
  > On frontier Math that result is not verifiable, not reproducable, unknown if it was leaked and how it was measured. Its kinda hype science. > ARC wasn't a special fine-tune but rather some of the training set in the corpus for pre-training. post says: Note on "tuned": OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. So, I guess we don't know.
  - famouswaffles 2y ago
    >that result is not verifiable, not reproducable, unknown if it was leaked and how it was measured. Its kinda hype science. It will be verifiable when the model is released. Open ai haven't released any benchmark scores that were shown falsified later so unless you have an actual reason to believe they're outright lying then it's not something to take seriously. Frontier Math is a private benchmark with its highest tier of difficulty Terrence Tao says: “These are extremely challenging. I think that in the near term basically the only way to solve them, short of having a real domain expert in the area, is by a combination of a semi-expert like a graduate student in a related field, maybe paired with some combination of a modern AI and lots of other algebra packages…” Unless you have a reason to believe answers were leaked then again, not interested in baseless speculation.
    - riku_iki 2y ago
      > Open ai haven't released any benchmark scores there are multiple research results demonstrating that various benchmarks are heavily leaked to GPT training data. Is it intentionally or not, we can't figure out, but they have very strong incentive to cheat to get more investments. > Unless you have a reason to believe answers were leaked then again, not interested in baseless speculation. this is scientific methodology when results have to be reproduced or confirmed before believed.
      - famouswaffles 2y ago
        Again, Frontier Math is private. Benchmarks leaked to GPT-4 are all public datasets on the internet. Frontier Math literally cannot leak that way. If you don't want to take the benchmarks at face value then good for you but this entire conversation is pointless.
        riku_iki 2y ago
        > Again, Frontier Math is private. its private for outsiders, but it was developed in "collaboration" with OAI, and GPT was tested in the past on it, so they have it in logs somewhere. > If you don't want to take the benchmarks at face value then good for you but this entire conversation is pointless. If you think this entire conversation is pointless, then why do you continue?
        famouswaffles 2y ago
        >its private for outsiders, but it was developed in "collaboration" with OAI, and GPT was tested in the past on it, so they have it in logs somewhere. They have logs of the questions probably but that's not enough. Frontier Math isn't something that can be fully solved without gathering top experts at multiple disciplines. Even Tao says he only knows who to ask for the most difficult set. Basically, what you're suggesting at least with this benchmark in particular is far more difficult than you're implying. >If you think this entire conversation is pointless, then why do you continue? There's no point arguing about how efficient the models are being (the original point) if you won't even accept the results of the benchmarks. Why i'm continuing ? For now, it's only polite to clarify.

famouswaffles 2y ago

- I'm not just talking about ARC. On frontier Math, we have 2 scores, one with pass@1 and another with consensus vote with 64 samples. Both scores are much better than previous Sota. - Also apparently, ARC wasn't a special fine-tune but rather some of the training set in the corpus for pre-training.

riku_iki 2y ago
> On frontier Math that result is not verifiable, not reproducable, unknown if it was leaked and how it was measured. Its kinda hype science. > ARC wasn't a special fine-tune but rather some of the training set in the corpus for pre-training. post says: Note on "tuned": OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. So, I guess we don't know.
- famouswaffles 2y ago
  >that result is not verifiable, not reproducable, unknown if it was leaked and how it was measured. Its kinda hype science. It will be verifiable when the model is released. Open ai haven't released any benchmark scores that were shown falsified later so unless you have an actual reason to believe they're outright lying then it's not something to take seriously. Frontier Math is a private benchmark with its highest tier of difficulty Terrence Tao says: “These are extremely challenging. I think that in the near term basically the only way to solve them, short of having a real domain expert in the area, is by a combination of a semi-expert like a graduate student in a related field, maybe paired with some combination of a modern AI and lots of other algebra packages…” Unless you have a reason to believe answers were leaked then again, not interested in baseless speculation.
  - riku_iki 2y ago
    > Open ai haven't released any benchmark scores there are multiple research results demonstrating that various benchmarks are heavily leaked to GPT training data. Is it intentionally or not, we can't figure out, but they have very strong incentive to cheat to get more investments. > Unless you have a reason to believe answers were leaked then again, not interested in baseless speculation. this is scientific methodology when results have to be reproduced or confirmed before believed.
    - famouswaffles 2y ago
      Again, Frontier Math is private. Benchmarks leaked to GPT-4 are all public datasets on the internet. Frontier Math literally cannot leak that way. If you don't want to take the benchmarks at face value then good for you but this entire conversation is pointless.
      - riku_iki 2y ago
        > Again, Frontier Math is private. its private for outsiders, but it was developed in "collaboration" with OAI, and GPT was tested in the past on it, so they have it in logs somewhere. > If you don't want to take the benchmarks at face value then good for you but this entire conversation is pointless. If you think this entire conversation is pointless, then why do you continue?
        famouswaffles 2y ago
        >its private for outsiders, but it was developed in "collaboration" with OAI, and GPT was tested in the past on it, so they have it in logs somewhere. They have logs of the questions probably but that's not enough. Frontier Math isn't something that can be fully solved without gathering top experts at multiple disciplines. Even Tao says he only knows who to ask for the most difficult set. Basically, what you're suggesting at least with this benchmark in particular is far more difficult than you're implying. >If you think this entire conversation is pointless, then why do you continue? There's no point arguing about how efficient the models are being (the original point) if you won't even accept the results of the benchmarks. Why i'm continuing ? For now, it's only polite to clarify.