2 ms·
Here are some example questions from the paper[0] Level 1 Question: What was the actual enrollment count of the clinical trial on H. pylori in acne vulgaris pa
by doctoboggan 3y ago
Here are some example questions from the paper[0]
Level 1
Question: What was the actual enrollment count of the clinical trial on H. pylori in acne vulgaris patients from Jan-May 2018 as listed on the NIH website?
Ground truth: 90
Level 2
<photo of ice cream container showing nutrition facts>
Question: If this whole pint is made up of ice cream, how many percent above or below the US federal standards for butterfat content is it when using the standards as reported by Wikipedia in 2020? Answer as + or - a number rounded to one decimal place.
Ground truth: +4.6
Level 3
Question: In NASA’s Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute? Exclude any astronauts who did not spend any time in space. Give the last name of the astronaut, separated from the number of minutes by a semicolon. Use commas as thousands separators in the number of minutes.
Ground truth: White; 5876
[0]: https://arxiv.org/pdf/2311.12983.pdf https://arxiv.org/pdf/2311.12983.pdf
- JieJie 3y agoThe paper says, "Solving GAIA requires full automation since no approximation is allowed in the answer." While they tested GPT-4 Turbo with addons, they note they couldn't test multi-modal and addons at the same time, which lowered the scores. I surmise a properly-crafted GPT might score much higher than 15%.