4 ms·
Ya, these questions seem too hard for an undergrad test or at least feel like they were only written in an age where you want to be adversarial to LLMs. Don't
by usaar333 4y ago
Ya, these questions seem too hard for an undergrad test or at least feel like they were only written in an age where you want to be adversarial to LLMs.
Don't get me wrong - I can give simple tests that LLMs fail and a college kid could easily pass -- but I don't think I'd come up with them unless I wanted to be adversarial to LLMs.
Example simple question that GPT4/Bard/etc. fail hard:
> The moose is behind the cat. The dog is in front of the moose. The moose is in front of the rat. The rat is behind the cat. The elk is in front of the dog.
What's a possible animal ordering?
- Kokouane 4y agoJust gave this prompt to GPT 4 and it gave what appears to be a correct answer: > rat, moose, cat, dog, elk Is this wrong somehow?
- usaar333 4y agohmm, maybe you got lucky? My answer is on ChatGPT4: A possible animal ordering, from front to back, would be: Elk, Dog, Moose, Cat, Rat (Which violates the moose being behind the cat) Another answer I get in a new session is: A possible animal ordering based on the given information is: Rat, Cat, Moose, Dog, Elk (which is the same as above just revered)