2 ms·
Some of the example prompts are unintentionally hilarious: > Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded
by Skeptology 2y ago
Some of the example prompts are unintentionally hilarious:
> Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.
LLMs are so intelligent they don't know that a "how many" question is answered with a number.
Also, something something Goodhart's law.
- famouswaffles 2y agoNo it's because, "Sure, here's your answer..." screws with evals.
- ClumsyPilot 2y agoWe need realistic tests - organise a pissup. here are WhatsApp texts of 10 people arguing about their dietary requirements, here are blurry screenshots of menus from nearest pubs and their prices. But no-one likes that one guy Joe Bogs, choose best place for everyone else and except him, so he doesn’t bother showing up
- agnishom 2y ago> LLMs are so intelligent they don't know that a "how many" question is answered with a number. I think this is to prevent the LLM from giving more details. The evaluation engine can presumably only check short exact answers.