3 ms·

Are they going to make the benchmark available so other LLMs can be compared?

• abhisuri97 2y ago
  https://github.com/openai/simple-evals/blob/main/simpleqa_eval.py https://github.com/openai/simple-evals/blob/main/simpleqa_ev...
  • seany62 2y ago
    Any way to see the actual questions and answers? Where can I find simple_qa_test_set.csv ?
    • sbierwagen 2y ago
      https://openaipublic.blob.core.windows.net/simple-evals/simple_qa_test_set.csv https://openaipublic.blob.core.windows.net/simple-evals/simp... The steps I took to find this link: 1) Look at simpleqa_eval.py. See that it loads "az://openaipublic/simple-evals/simple_qa_test_set.csv" Hmm, some weird vendored protocol. 2) I don't feel like digging through bf.BlobFile() to figure out how it downloads files and I certainly don't want to generate an API key. Cross fingers and do a Bing web search for "az://openaipublic" 3) That leads me to https://stackoverflow.com/questions/76106366/how-to-use-tiktoken-in-offline-mode-computer https://stackoverflow.com/questions/76106366/how-to-use-tikt... Ah ha, this answer has the link https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken https://openaipublic.blob.core.windows.net/encodings/cl100k_... which automatically downloads a file. 4) Poke the relevant parts of the az:// link into this link, and a csv appears.
      • _jonas 2y ago
        It's easier to find the data now, I've run some benchmarks on it. Great to see OpenAI open-sourcing datasets like this!