2 ms·
That seems similar to a ~7th grade reading comprehension question, if all the facts where at hand. Out of curiosity, if anyone knows, what's SOTA for how well
by ethbr1 2y ago
That seems similar to a ~7th grade reading comprehension question, if all the facts where at hand.
Out of curiosity, if anyone knows, what's SOTA for how well LLMs actually parse (English) grammar? In the way they're looking at the prompt.
A lot of correctness to the challenge questions seems to be identifying key phrases and requests. I.e. reading comprehension.
And multi-step tool use requires a higher bar than straight summarization, as one must more particularly differentiate between alternative information to focus on.
- adtac 2y agoThe question above was not preceded by anything; that was the whole question. The facts are at hand in the sense that you have the internet and you're allowed to use it. The hard part is knowing what to search and recognising the answer when you see it. This is much harder than any 7th grade comprehension test I've done :)