4 ms·

Meta: Gaia - A Benchmark for General AI Assistants

36 points by georgehill 3y ago

andsoitis 3y ago
> We release our questions while retaining answers to 300 of them to power a leader-board available at this https URL. Nothing on the leaderboard: https://huggingface.co/gaia-benchmark https://huggingface.co/gaia-benchmark
- brucethemoose2 3y ago
  Good, but honestly they should have retained the questions too! And yeah, the leaderboard appears to be down with a nice Python stacktrace.
pikseladam 3y ago
interesting take. another milestone will be achieved when GAIA defeated. who will be the first? :)
doctoboggan 3y ago
Here are some example questions from the paper[0] Level 1 Question: What was the actual enrollment count of the clinical trial on H. pylori in acne vulgaris patients from Jan-May 2018 as listed on the NIH website? Ground truth: 90 Level 2 <photo of ice cream container showing nutrition facts> Question: If this whole pint is made up of ice cream, how many percent above or below the US federal standards for butterfat content is it when using the standards as reported by Wikipedia in 2020? Answer as + or - a number rounded to one decimal place. Ground truth: +4.6 Level 3 Question: In NASA’s Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute? Exclude any astronauts who did not spend any time in space. Give the last name of the astronaut, separated from the number of minutes by a semicolon. Use commas as thousands separators in the number of minutes. Ground truth: White; 5876 [0]: https://arxiv.org/pdf/2311.12983.pdf https://arxiv.org/pdf/2311.12983.pdf
- JieJie 3y ago
  The paper says, "Solving GAIA requires full automation since no approximation is allowed in the answer." While they tested GPT-4 Turbo with addons, they note they couldn't test multi-modal and addons at the same time, which lowered the scores. I surmise a properly-crafted GPT might score much higher than 15%.
rkwasny 3y ago
We need to double check this questions manually first if the answers are actually correct
- gregmialz 3y ago
  First author here, half of the crafting process actually consists in double checking the questions
Thomjazz 3y ago
Hi some authors of the work here, thanks a lot for sharing the paper, it's been quite some time in the work and we're super happy to share it with the world. A short note on some of the reasons we decided to go with openly-sharing the questions instead of holding them back (which was another option we contemplated): - with closed-models we need to send the questions through an external AI anyway so a full privacy of the test set is not possible in general unless the leaderboard is restricted to open models (would be quite restrictive) - also, the benchmark contain a limited number of questions which are non-obvious and take a significant time for human reviewers to solve. We thus don't expect the dataset to become training material for models and to lead to having model over-fitting on the benchmark pattern in the traditional sense that happened with larger benchmark datasets including a training split. This benchmark is generally closer in philosophy to small, hand crafted benchmark datasets, like HumanEval for instance has been for code models.

36 points by georgehill 3y ago

> We release our questions while retaining answers to 300 of them to power a leader-board available at this https URL. Nothing on the leaderboard: https://huggingface.co/gaia-benchmark https://huggingface.co/gaia-benchmark

brucethemoose2 3y ago
Good, but honestly they should have retained the questions too! And yeah, the leaderboard appears to be down with a nice Python stacktrace.

pikseladam 3y ago

interesting take. another milestone will be achieved when GAIA defeated. who will be the first? :)

doctoboggan 3y ago

Here are some example questions from the paper[0] Level 1 Question: What was the actual enrollment count of the clinical trial on H. pylori in acne vulgaris patients from Jan-May 2018 as listed on the NIH website? Ground truth: 90 Level 2 <photo of ice cream container showing nutrition facts> Question: If this whole pint is made up of ice cream, how many percent above or below the US federal standards for butterfat content is it when using the standards as reported by Wikipedia in 2020? Answer as + or - a number rounded to one decimal place. Ground truth: +4.6 Level 3 Question: In NASA’s Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute? Exclude any astronauts who did not spend any time in space. Give the last name of the astronaut, separated from the number of minutes by a semicolon. Use commas as thousands separators in the number of minutes. Ground truth: White; 5876 [0]: https://arxiv.org/pdf/2311.12983.pdf https://arxiv.org/pdf/2311.12983.pdf

JieJie 3y ago
The paper says, "Solving GAIA requires full automation since no approximation is allowed in the answer." While they tested GPT-4 Turbo with addons, they note they couldn't test multi-modal and addons at the same time, which lowered the scores. I surmise a properly-crafted GPT might score much higher than 15%.

rkwasny 3y ago

We need to double check this questions manually first if the answers are actually correct

gregmialz 3y ago
First author here, half of the crafting process actually consists in double checking the questions

Thomjazz 3y ago

Hi some authors of the work here, thanks a lot for sharing the paper, it's been quite some time in the work and we're super happy to share it with the world. A short note on some of the reasons we decided to go with openly-sharing the questions instead of holding them back (which was another option we contemplated): - with closed-models we need to send the questions through an external AI anyway so a full privacy of the test set is not possible in general unless the leaderboard is restricted to open models (would be quite restrictive) - also, the benchmark contain a limited number of questions which are non-obvious and take a significant time for human reviewers to solve. We thus don't expect the dataset to become training material for models and to lead to having model over-fitting on the benchmark pattern in the traditional sense that happened with larger benchmark datasets including a training split. This benchmark is generally closer in philosophy to small, hand crafted benchmark datasets, like HumanEval for instance has been for code models.