4 ms·
I didn't know ARC-AGI had been "beaten" by o3. What are the next challenges that frontier models like o1/o3 are faced with?
by knowaveragejoe 2y ago
I didn't know ARC-AGI had been "beaten" by o3. What are the next challenges that frontier models like o1/o3 are faced with?
- chriscappuccio 2y agoo1 did terrible. o3 did well on arc-agi-pub (public training data) but hasn't passed the private test yet.
- lucianbr 2y agoIs the test still private once it has been run? If you call the OpenAI API and send it some data, OpenAI has access to the data. Did the benchmaker run the models locally somehow?
- vbarrielle 2y agoThe private test is supposed to be run by the ARC-AGI organization themselves, without network access. That's why o3 has not been run against it yet. Not sure if it will be possible either, depends on what OpenAI is prepared to do about it.