3 ms·

Dataset: http://openaipublic.blob.core.windows.net/simple-evals/simple_qa_test_set.csv http://openaipublic.blob.core.windows.net/simple-evals/simpl...

by chgo1 2y ago

YetAnotherNick 2y ago
First few questions for those who don't care to download. Most just seem to be about niche facts: Who received the IEEE Frank Rosenblatt Award in 2010? Who was awarded the Oceanography Society's Jerlov Award in 2018? What's the name of the women's liberal arts college in Cambridge, Massachusetts? In whose honor was the Leipzig 1877 tournament organized? According to Karl Küchler, what did Empress Elizabeth of Austria's favorite sculpture depict, which was made for her villa Achilleion at Corfu? How much money, in euros, was the surgeon held responsible for Stella Obasanjo's death ordered to pay her son?
- chaxor 2y ago
  Also importantly, they do have a 'not attempted' or 'do not know' type of response, though how it is used is not really well discussed in the article. As it has been for decades now, the 'Nan' type of answer in NLP is important, adds great capability, and is often glossed over.
  - bcherry 2y ago
    a little glossed over, but they do point out that most important improvement o1 has over gpt-4o is not it's "correct" score improving from 38% to 42% but actually it's "not attempted" going from 1% to 9%. The improvement is even more stark for o1-mini vs gpt-4o-mini: 1% to 28%. They don't really describe what "success" would look like but it seems to me like the primary goal is to minimize "incorrect", rather than to maximize "correct". the mini models would get there by maximizing "not attempted" with the larger models having much higher "correct". Then both model sizes could hopefully reach 90%+ "correct" when given access to external lookup tools.
- nilstycho 2y ago
  > What's the name of the women's liberal arts college in Cambridge, Massachusetts? Wait, what is the correct answer? “Radcliffe College”?
  - YetAnotherNick 2y ago
    Yes
    - jefftk 2y ago
      Not surprising that this would be on a list of questions at least one model got wrong, since I think the real answer is "there isn't one anymore, but from 1879 to 1999 the answer would have been Radcliffe College".
      - nilstycho 2y ago
        Yes, that would be my preferred answer!

Dataset: http://openaipublic.blob.core.windows.net/simple-evals/simple_qa_test_set.csv http://openaipublic.blob.core.windows.net/simple-evals/simpl...

by chgo1 2y ago

Dataset: http://openaipublic.blob.core.windows.net/simple-evals/simple_qa_test_set.csv http://openaipublic.blob.core.windows.net/simple-evals/simpl...

YetAnotherNick 2y ago

First few questions for those who don't care to download. Most just seem to be about niche facts: Who received the IEEE Frank Rosenblatt Award in 2010? Who was awarded the Oceanography Society's Jerlov Award in 2018? What's the name of the women's liberal arts college in Cambridge, Massachusetts? In whose honor was the Leipzig 1877 tournament organized? According to Karl Küchler, what did Empress Elizabeth of Austria's favorite sculpture depict, which was made for her villa Achilleion at Corfu? How much money, in euros, was the surgeon held responsible for Stella Obasanjo's death ordered to pay her son?

chaxor 2y ago
Also importantly, they do have a 'not attempted' or 'do not know' type of response, though how it is used is not really well discussed in the article. As it has been for decades now, the 'Nan' type of answer in NLP is important, adds great capability, and is often glossed over.
- bcherry 2y ago
  a little glossed over, but they do point out that most important improvement o1 has over gpt-4o is not it's "correct" score improving from 38% to 42% but actually it's "not attempted" going from 1% to 9%. The improvement is even more stark for o1-mini vs gpt-4o-mini: 1% to 28%. They don't really describe what "success" would look like but it seems to me like the primary goal is to minimize "incorrect", rather than to maximize "correct". the mini models would get there by maximizing "not attempted" with the larger models having much higher "correct". Then both model sizes could hopefully reach 90%+ "correct" when given access to external lookup tools.
nilstycho 2y ago
> What's the name of the women's liberal arts college in Cambridge, Massachusetts? Wait, what is the correct answer? “Radcliffe College”?
- YetAnotherNick 2y ago
  Yes
  - jefftk 2y ago
    Not surprising that this would be on a list of questions at least one model got wrong, since I think the real answer is "there isn't one anymore, but from 1879 to 1999 the answer would have been Radcliffe College".
    - nilstycho 2y ago
      Yes, that would be my preferred answer!