4 ms·
Yeah this seems like a trick question that can be interpreted multiple ways, especially if english isn't your native language. This is exactly why I prefer the
by wildpeaks 4y ago
Yeah this seems like a trick question that can be interpreted multiple ways, especially if english isn't your native language.
This is exactly why I prefer the precision of code over natural language for inputs (and why Google's botsplaining of "what you really meant to search" instead of just searching exactly the query as-is is so frustrating).
In a test, this seems like the kind of question they use to justify not giving a perfect grade because whatever answer you pick, they can claim it was the other one because both interpretations are reasonable.
- antihipocrat 4y agoI just tested it with clearer language... and the response is incorrect: Question: There are 5 dogs in a room right now. The previous day an additional dog was purchased and added to the room. How many dogs are in the room right now? Answer: There are 6 dogs in the room right now. The statement says that the previous day an additional dog was purchased and added to the room, so the total number of dogs in the room right now is 5 + 1 = 6. Edit: Ok a comment below pointed out that the question is still unclear. See below for the fix and the response. It is still incorrect: Q: There are 5 dogs in a room on 1 January 2023. The previous day an additional dog was purchased and added to the room. How many dogs are in the room on 1 January 2023? A: There are 6 dogs in the room on 1 January 2023. The previous day, one additional dog was purchased and added to the room, which means there are 5 dogs + 1 additional dog = 6 dogs in the room on 1 January 2023.
- mpoteat 4y agoReading the question, I thought the answer was six as well. Such a question is so misleading that I would hope that an AI answers as it has in this case, because not all problems are fully logically specified, and you need to remain robust against the question being posed incorrectly. In other words, I would posit the probability of you having meant for the answer to be 5 low, because the question itself becomes trivial. For a system that needs to deal with many people asking questions, this robustness is helpful in my view. In other words, it's not totally clear that time has not passed between sentences in your prompt. So the second instance of "now" could be a different "now".
- antihipocrat 4y agoGood point, I added a clearer version of the question to my comment above. It still has trouble with it.
- Jensson 4y agoReading and understanding exactly what they are asking for is a part of the test, adding superfluous information to problems is perfectly valid to test that students actually understand instead of trying to pattern match to solutions they memorized.
- heavyset_go 4y agoExcept the AI is not responding this way because it thinks the probability of the OP having meant something else was high. It answered the way it did because it is incapable of understanding the question it was asked, along with any question it is asked. The "robustness" you're perceiving in this case is just a mere coincidence, and doesn't reflect an aptitude for answering poorly phrased questions, but it does reflect a fundamental problem with these systems.
- whakim 4y agoI think the word "additional" is pretty confusing here, because it begs the question of "additional to what?" (ChatGPT still doesn't get it right if you remove "additional")
- junon 4y agoThat doesn't matter. The question itself assets as fact that there are X objects right now. Then it provides a superfluous fact. Then it asks, directly, "how many objects right now?" in which the answer is always X.
- deleted 4y ago[deleted]
- zug_zug 4y agoI think you have a good point here -- except did you ask it using "collective chain of thought" ? The whole premise of this link is that if you ask the question in a special way to make openAI simulate multiple agents it tends to do better. I'd try myself but openAI seems at-capacity.