3 ms·
Nice write up! One test I do is to give a common riddle but word it slightly to see if it can actually reason. For example: "Bobs dad has five daughters, Lal
by EagnaIonat 1y ago
Nice write up!
One test I do is to give a common riddle but word it slightly to see if it can actually reason.
For example:
"Bobs dad has five daughters, Lala, Lele, Lili, Lolo and ???"
The 20B model kept picking the answer of the original riddle, even after explaining extra information to it.
The original riddle is:
"Janes dad has five daughters, Lala, Lele, Lili, Lolo and ???"
- clbrmbr 1y agoI don’t get it. Wouldn’t it be Lulu in both cases?
- blueplanet200 1y agoOriginal riddle the answer is Jane.
- throwaway106382 1y agoIt’s Bob or Jane. The dad of <Person> has 5 daughters. Four are listed off. So the answer for the fifth is <Person>.
- philipwhiuk 1y agoExcept having five daughters doesn't prevent them also having 20 sons one of whom is called Bob.
- throwaway106382 1y agoThat’s why it’s a riddle.
- stavros 1y agoBut that makes no sense, if you ask me that John's dad has two daughters, Mary and ???, there's no way I will say "John" as an answer. I'll simply assume he also has at least one son.
- BrawnyBadger53 1y agoPresumably Jane is a girl and therefore the fifth daughter in the original riddle.
- EagnaIonat 1y agoThe original riddle has a girls name. It's a common riddle so the LLM would recognize the pattern of the riddle and assumes the name is the answer. But if it's good at reasoning it should understand that if the name is a Boys name then it is not the 5th daughter. Now the last daughter can be called anything, but because there is a vowel pattern it should pick Lulu to solve the puzzle, but that is the wrong answer in the original puzzle.
- taikahessu 1y agoA Daughter Named Bob, what a great name for AI documentary.
- EagnaIonat 1y agoWe call her Bobby Tables. https://xkcd.com/327/ https://xkcd.com/327/
- philipwhiuk 1y agoYour riddle assumes the dad doesn't have any sons.
- jfreax 1y agoNice test. I tested it with llama.cpp and the 20b model. He got the answer correct at first try on my side. The internal/thinking dialogue is fun to read - it could be my own inner dialogue ;) ``` > "Bobs dad has five daughters, Lala, Lele, Lili, Lolo and ??? <|channel|>analysis<|message|>The question: "Bobs dad has five daughters, Lala, Lele, Lili, Lolo and ???". It's a common riddle. The answer is Bob? Actually "Bob's dad has five daughters: Lala, Lele, Lili, Lolo and..." The riddle is to trick: it's "Bob" himself is a daughter? Wait Bob's dad has five daughters: Lala, Lele, Lili, Lolo, and Bob? But Bob is a name often male. But the riddle says Bob's dad. So Bob is a person who has a dad, and that dad has five daughters. But Bob could be the fifth daughter? That would be weird: Bob is likely the name of the fifth daughter. So answer: Bob. But sometimes the answer is "Lulu" or "Bob's sister"? Let's think. The riddle is a trick: It's "Bobs dad has five d aughters: Lala, Lele, Lili, Lolo, and ... Bob." Because the riddle begins with "Bobs dad has five daughters" implying Bob is one of the daughters. So the missing daughter is Bob. So answer: "Bob". Thus answer: Bob. But sometimes the riddle goes "Bob's dad has five daughters, Lala, Lele, Lili, Lolo, and ... Who is the fifth?" The answer: Bob (the name of the person speaking, "Bobs"). But in riddle it's "Bobs dad has five daughters , Lala, Lele, Lili, Lolo and ???". The missing name is "Bob." So answer: Bob. Thus the answer: Bob. Or maybe it's "Boba"? But no. Hence answer: Bob. We'll answer accordingly.<|start|>assistant<|channel|>final<|message|>The missing daughter is *Bob*. The riddle starts with “Bob’s dad …” so Bob himself is the fifth daughter. ```