4 ms·

The paper evaluates itself on the GAIA benchmark and it was my first time hearing about it, so I tried to evaluate myself as a human. Here's a level 3 question

by adtac 2y ago

The paper evaluates itself on the GAIA benchmark and it was my first time hearing about it, so I tried to evaluate myself as a human. Here's a level 3 question from the GAIA paper (level 3 = hardest): >In NASA’s Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute? Exclude any astronauts who did not spend any time in space. Give the last name of the astronaut, separated from the number of minutes by a semicolon. Use commas as thousands separators in the number of minutes. I timed myself solving the problem. It took me 9 minutes, 5 Google searches, 14 web pages, multiple Ctrl+F in these pages and 1 calculator use to figure out the answer. DynaSaur seems to have a 10% to 20% success rate at this level. Try for yourself. This is one of the few empirically grounded reference levels for how far we are from AGI.

ethbr1 2y ago
That seems similar to a ~7th grade reading comprehension question, if all the facts where at hand. Out of curiosity, if anyone knows, what's SOTA for how well LLMs actually parse (English) grammar? In the way they're looking at the prompt. A lot of correctness to the challenge questions seems to be identifying key phrases and requests. I.e. reading comprehension. And multi-step tool use requires a higher bar than straight summarization, as one must more particularly differentiate between alternative information to focus on.
- adtac 2y ago
  The question above was not preceded by anything; that was the whole question. The facts are at hand in the sense that you have the internet and you're allowed to use it. The hard part is knowing what to search and recognising the answer when you see it. This is much harder than any 7th grade comprehension test I've done :)

by adtac 2y ago

ethbr1 2y ago

That seems similar to a ~7th grade reading comprehension question, if all the facts where at hand. Out of curiosity, if anyone knows, what's SOTA for how well LLMs actually parse (English) grammar? In the way they're looking at the prompt. A lot of correctness to the challenge questions seems to be identifying key phrases and requests. I.e. reading comprehension. And multi-step tool use requires a higher bar than straight summarization, as one must more particularly differentiate between alternative information to focus on.

adtac 2y ago
The question above was not preceded by anything; that was the whole question. The facts are at hand in the sense that you have the internet and you're allowed to use it. The hard part is knowing what to search and recognising the answer when you see it. This is much harder than any 7th grade comprehension test I've done :)