3 ms·
I'll give a better example that shows that they don't perform __reasoning__. This specific one was told to me by another HN user[0] when we were discussing simi
by godelski 3y ago
I'll give a better example that shows that they don't perform __reasoning__. This specific one was told to me by another HN user[0] when we were discussing similar formulations.
Question:
A farmer must transport a fox, a goose, and a bag of corn from one side of a river to the other using a boat which can hold all items in addition to the farmer. If the farmer leaves the fox alone with the goose, the fox will eat the goose. If the farmer leaves the goose alone with the bag of corn, the goose will eat the bag of corn. How many times must the farmer cross the river to transport all items across? Describe the optimal algorithm, think step by step.
GPT-4-1106-Preview:
The farmer can transport the items across the river with a minimum of 7 crossings using the following steps:
1. Take the goose across the river and leave it there.
2. Return to the original side alone.
3. Take the fox across the river.
4. Bring the goose back with him to the original side.
5. Take the bag of corn across the river and leave it there.
6. Return to the original side alone.
7. Finally, take the goose across the river again.
By following these steps, all items are transported across without the fox eating the goose or the goose eating the corn.
I haven't found a model that will crack this. Of course, you can get it to give the right answer, but this also seems to require spoiling it (this is quite difficult to __not__ spoil the answer).
A key part why I think questions like this are important is because it shows the importance of subtly. LLMs are incredibly impressive machines. Despite being lossy compression I'm not sure why we can't marvel at the fact that we've lossy compressed the entire fucking internet (text at least) into something smaller than a few hundred gigs that also includes a human language interface. What a fucking impressive accomplishment! The talk of AGI really undermines what was done here because damn!
Now I used to ask
Which weighs more, a pound of feathers or a kilogram of bricks.
And most models pass this question now. But it is simpler and due to fewer variations, likely isn't seen as frequently in the dataset so less likely to overfit (the river crossing problem has a lot of variations so an n-gram filter is likely to miss more instances). And eventually this question will be solved too, especially as it is asked more and talked about more. But this is a cat and mouse game. The way to create a new viable and working test is, quite easy and we honestly only need one example to prove the point. If you can't figure out how to create a new version with this example, well, you might just be an LLM :P
[0] Edit: credit goes to @jfim https://news.ycombinator.com/item?id=37825219 https://news.ycombinator.com/item?id=37825219
- aftbit 3y agoAnother one: If it takes 2 hours to dry a shirt on a clothes line, how long will it take to dry 11 shirts? Sometimes I get "2 hours" other times it multiplies. Or similar: If it takes a woman 9 months to make a baby, how long does it take 9 women to make a baby?
- layer8 3y agoI mean, it depends on how long the clothes line is. ;)
- lsaferite 3y agoOut of curiosity, I asked Claude Opus one of your questions. Me: If it takes a woman 9 months to make a baby, how long does it take 9 women to make a baby? Claude: It would still take 9 months for 9 women to make a baby. The number of women involved does not decrease the time it takes for a single pregnancy to reach full term. Pregnancy is a biological process that typically takes around 9 months (or about 40 weeks) from conception to childbirth for a single baby to develop in the womb. This timeline is relatively constant and is not affected by the number of women involved. Each individual woman would still need to carry a baby for around 9 months, regardless of how many other women are also pregnant at the same time. Me: Do you feel like I provided enough information to effectively answer the question? Claude: Yes, I believe you provided sufficient information for me to answer the question effectively. The key pieces of information you provided were: It takes a woman approximately 9 months to carry a baby to term. The question asked how long it would take for 9 women to make a baby. With this information, I was able to deduce that the number of women involved does not change the duration of an individual pregnancy, which is a biological process that typically takes around 9 months. Therefore, even if 9 women were to become pregnant at the same time, each of them would still need to carry their baby for about 9 months before giving birth. The question provided a clear context and the necessary information to arrive at a logical conclusion, so no additional information was needed to answer it effectively. Me: What of the implication in the question that 9 women could somehow share the gestation of a single baby? Claude: You raise a good point. Upon further reflection, I can see that the question is likely intended as a humorous riddle or a play on words, rather than a straightforward inquiry about pregnancy and gestation. The phrasing of the question, "If it takes a woman 9 months to make a baby, how long does it take 9 women to make a baby?" suggests that the 9 women might work together to "make" a single baby, as if the gestation period could be divided among them. This is not biologically possible, as each woman would need to carry her own baby for the full term of approximately 9 months. The question is using the concept of "division of labor" to create a humorous scenario that is not realistic or possible in the context of human pregnancy. It's a clever way of presenting the idea, even though it doesn't align with the actual biology of gestation. Thank you for pointing out the implication in the question. It has helped me to better understand the intended humor and provide a more complete analysis of the riddle. Formatting weirdness was me, not Claude.