4 ms·

> This is a trivial question. There's one correct answer and the reasoning to get there takes one step: the car needs to be at the car wash, so you drive. I do

by layer8 8mo ago

> This is a trivial question. There's one correct answer and the reasoning to get there takes one step: the car needs to be at the car wash, so you drive. I don’t think it’s that easy. An intelligent mind will wonder why the question is being asked, whether they misunderstood the question, or whether the asker misspoke, or some other missing context. So the correct answer is neither “walk” nor “drive”, but “Wat?” or “I’m not sure I understand the question, can you rephrase?”, or “Is the vehicle you would drive the same as the car that you want to wash?”, or “Where is your car currently located?”, and so on.

felix089 8mo ago
That's a fair point, but if you would see it as a riddle, which I don't really think it is, and you had to answer either or, I'd still assume it's most logical to chose drive isn't it?
- layer8 8mo ago
  I don’t agree that the question as written would qualify as a riddle. If anything, the riddle is what the intention of the asker is. One can always ask stupid questions with an artificially limited set of answering options; that doesn’t mean it makes sense.
  - felix089 8mo ago
    I don't think it qualifies as a stupid question either, it does make sense
    - rerdavies 8mo ago
      It is TOTALLY a stupid question, because OBVIOUSLY you should drive. It is based on the false premise that there is actually a choice. If somebody were to sincerely ask me this question, actually believing that walking was an option, I'm not sure I could resist the temptation to say "walk", just to see what happens next. Only slightly evil, because the worst-case consequences are an unnecessary 100m walk. I think I could get that past an ethics committee, if I wanted to run an experiment to see what percentage of human responders would ACTUALLY walk to the car wash.
Night_Thastus 8mo ago
I agree. If the LLM were truly an intelligence, it would be able to ask about this nonsense question. It would be able to ask "Why is walking even an option? Can you please explain how you imagine that would work? Do you mean hand-washing the car at home, instead?" (etc, etc) Real people can ask for clarification when things are ambiguous or confusing. Once something is clarified, they can work that into their understanding of how someone communicates about a given topic. An LLM can't.
- CamperBob2 8mo ago
  Gemini's responses come very close to doing that when they make fun of the question (see other posts in the thread). If the model had been RL'ed to ask follow-up questions, it seems likely that it would meet your criterion.
- umpalumpaaa 8mo ago
  LLMs like the ones from Claude can ask questions and even have you pick from multiple choices or provide your own answer…
- rerdavies 8mo ago
  And the corollary: if LLMs were truly intelligent, they would also be able to respond to such questions sarcastically.
  - CamperBob2 8mo ago
    Which Gemini does...?
buu700 8mo ago
Same energy: https://youtu.be/8ERyTfm1Dxw https://youtu.be/8ERyTfm1Dxw
nozzlegear 8mo ago
I think most people would say "drive?" and wonder when the punchline is coming, but (IMO) I don't think they'd start asking for clarification right away.
- 1718627440 8mo ago
  You can ask with your eyes, tone or face expression. That "?" at the end of "drive?" is a request for clarification or a punchline.
BrenBarn 8mo ago
The reason that those questions are asked, though, is that the answer to the actual question is obvious, so a human will start to wonder if it's some kind of trick.
- layer8 8mo ago
  The answer wasn’t obvious to me, it was more like “parse error”.
kayge 8mo ago
Yep, just a little more context and all/most of the models would do much better. And sure, most average+ intelligence adults whose first language is English (probably) don't need this, but they're not the target audience for the instructions :) "The 'car wash' is a building I need to drive through." or "The 'car wash' is a bottle of cleaning fluid that I left at the end of my driveway." https://i5.walmartimages.com/seo/Rain-x-Foaming-Car-Wash-Concentrate-100oz-62091_a2ee8b0c-392f-4242-8dcd-ad9f5a11298c.7bb678ec6be6c313addf3e8964940eba.jpeg https://i5.walmartimages.com/seo/Rain-x-Foaming-Car-Wash-Con...
ranger_danger 8mo ago
Agreed. It's also possible that "car wash" merely refers to soap they might use to do it themselves, and they're only going to buy it and then wash the car themselves at home. Imagine the same question but substitute "wash" for "wax" and it makes even more sense IMO.
redwood 8mo ago
Thank you for saying this. It reminds me of class tests where you always had to wonder if something was a trick question and you never really knew... it was always after the teacher. Which frankly is fine in open-ended questions where you can explain your rationale or how different interpretations would lead you to different paths but a terrible situation when it comes to multiple choice. I remember being very frustrated by those
mattclarkdotnet 8mo ago
Fully agreed. It’s like the bad phrasings of the Monty Hall problem where it’s not made clear Monty actively chooses to show a goat.
Nition 8mo ago
It highlights a general problem with LLMs, that they always jump to answering, whereas humans will often ask clarifying questions first.
- user_7832 8mo ago
  I wonder if anyone has any research on this field. I've often seen this myself (too often) where LLMs make assumptions and run off with the wrong thing. "This is how you do <absolutely unrelated thing>" or "This is why <thing that actually exists already> is impossible!". Ffs man, just ask for info! A human wouldn't need to - they'd get the context - but LLMs apparently don't?
  - magackame 8mo ago
    Don't people do this too all the time?
- micw 8mo ago
  Maybe that's a bias from training data. I would assume that most documents skip the "clarifying the question/scope" part of reasoning. Imagine a scientific text or even a book. Most will start with a clear context/scope. Either with a thesis or a well defined question or (in case of a book) with a story. Texts that start with a question that first needs to be refined are probably rare.
red75prime 8mo ago
An intelligent mind should keep a balance. U: The car wash is 50m away. Should I walk or should I drive? A: Do you need to wash your car? U: Yep. A: Is the car yours? U: Yep. A: Is your car nearby? U: Yes. A: Do you need to walk to your car? U: No. A: Do you know if the car wash is currently working? U: Yes. A: Does it work? U: YES! A: Do you have a way to pay for the wash? U: YES! A: On you? U: YES! A: Is your clothing appropriate for the occasion? U: Argh...
jiggawatts 8mo ago
This reminds me of a Uni exam that was soooo broken that answering “correctly” entailed guessing how exactly the professor designing the questions misunderstood the topic of his own lectures.
- rerdavies 8mo ago
  An interesting parallel to that is the "What's the next number in this sequence?" sort of questions. If four numbers are provided, one can calculate the coefficients of a a quartic polynomial, for x values of 0, 1, 2 and 3, and then solve for x=4. Which does indeed provide a defensible "next number". And by similar reasoning, there are an infinite number of answers to this question. Even worse. You could in fact provide any number as an answer, because there is always a quintic polynomial that fits the four initial numbers AND your arbitrary fifth number. So these questions are actually not about what the next number is, but trying to imagine what the person who set the question thought was a "cool" answer, for some curious definition of "cool", for some person who isn't smart enough to realize that the premise on which the question is based is flawed.
- 1718627440 8mo ago
  Are you not allowed to ask the professor questions? We are, and it is not to seldom that the professor then walks to the black board and updates the question.
  - jiggawatts 8mo ago
    It was an examination with 300 students in a giant hall, overseen by university staff, not the individual professors. So many people complained that they did eventually fetch him to come and clarify (correct) the questions. I didn’t have the patience to wait for him to turn up, so I simply provided a matrix of solutions for every possible combination of potential original intent… with note next to it saying that anything other than a 100% mark will be met with official complaints about his lack of due diligence.
easyThrowaway 8mo ago
It feels more like a question on english linguistic conventions than logic. If someone asked me the same question and I wanted to give a smartass reply, I'd tell them "You want to wash your car, good to know. Now, about your question, unless you tell me where you wanna go I can't really help you".

by layer8 8mo ago

felix089 8mo ago

That's a fair point, but if you would see it as a riddle, which I don't really think it is, and you had to answer either or, I'd still assume it's most logical to chose drive isn't it?

layer8 8mo ago
I don’t agree that the question as written would qualify as a riddle. If anything, the riddle is what the intention of the asker is. One can always ask stupid questions with an artificially limited set of answering options; that doesn’t mean it makes sense.
- felix089 8mo ago
  I don't think it qualifies as a stupid question either, it does make sense
  - rerdavies 8mo ago
    It is TOTALLY a stupid question, because OBVIOUSLY you should drive. It is based on the false premise that there is actually a choice. If somebody were to sincerely ask me this question, actually believing that walking was an option, I'm not sure I could resist the temptation to say "walk", just to see what happens next. Only slightly evil, because the worst-case consequences are an unnecessary 100m walk. I think I could get that past an ethics committee, if I wanted to run an experiment to see what percentage of human responders would ACTUALLY walk to the car wash.

Night_Thastus 8mo ago

I agree. If the LLM were truly an intelligence, it would be able to ask about this nonsense question. It would be able to ask "Why is walking even an option? Can you please explain how you imagine that would work? Do you mean hand-washing the car at home, instead?" (etc, etc) Real people can ask for clarification when things are ambiguous or confusing. Once something is clarified, they can work that into their understanding of how someone communicates about a given topic. An LLM can't.

CamperBob2 8mo ago
Gemini's responses come very close to doing that when they make fun of the question (see other posts in the thread). If the model had been RL'ed to ask follow-up questions, it seems likely that it would meet your criterion.
umpalumpaaa 8mo ago
LLMs like the ones from Claude can ask questions and even have you pick from multiple choices or provide your own answer…
rerdavies 8mo ago
And the corollary: if LLMs were truly intelligent, they would also be able to respond to such questions sarcastically.
- CamperBob2 8mo ago
  Which Gemini does...?

buu700 8mo ago

Same energy: https://youtu.be/8ERyTfm1Dxw https://youtu.be/8ERyTfm1Dxw

nozzlegear 8mo ago

I think most people would say "drive?" and wonder when the punchline is coming, but (IMO) I don't think they'd start asking for clarification right away.

1718627440 8mo ago
You can ask with your eyes, tone or face expression. That "?" at the end of "drive?" is a request for clarification or a punchline.

BrenBarn 8mo ago

The reason that those questions are asked, though, is that the answer to the actual question is obvious, so a human will start to wonder if it's some kind of trick.

layer8 8mo ago
The answer wasn’t obvious to me, it was more like “parse error”.

kayge 8mo ago

Yep, just a little more context and all/most of the models would do much better. And sure, most average+ intelligence adults whose first language is English (probably) don't need this, but they're not the target audience for the instructions :) "The 'car wash' is a building I need to drive through." or "The 'car wash' is a bottle of cleaning fluid that I left at the end of my driveway." https://i5.walmartimages.com/seo/Rain-x-Foaming-Car-Wash-Concentrate-100oz-62091_a2ee8b0c-392f-4242-8dcd-ad9f5a11298c.7bb678ec6be6c313addf3e8964940eba.jpeg https://i5.walmartimages.com/seo/Rain-x-Foaming-Car-Wash-Con...

ranger_danger 8mo ago

Agreed. It's also possible that "car wash" merely refers to soap they might use to do it themselves, and they're only going to buy it and then wash the car themselves at home. Imagine the same question but substitute "wash" for "wax" and it makes even more sense IMO.

redwood 8mo ago

Thank you for saying this. It reminds me of class tests where you always had to wonder if something was a trick question and you never really knew... it was always after the teacher. Which frankly is fine in open-ended questions where you can explain your rationale or how different interpretations would lead you to different paths but a terrible situation when it comes to multiple choice. I remember being very frustrated by those

mattclarkdotnet 8mo ago

Fully agreed. It’s like the bad phrasings of the Monty Hall problem where it’s not made clear Monty actively chooses to show a goat.

Nition 8mo ago

It highlights a general problem with LLMs, that they always jump to answering, whereas humans will often ask clarifying questions first.

user_7832 8mo ago
I wonder if anyone has any research on this field. I've often seen this myself (too often) where LLMs make assumptions and run off with the wrong thing. "This is how you do <absolutely unrelated thing>" or "This is why <thing that actually exists already> is impossible!". Ffs man, just ask for info! A human wouldn't need to - they'd get the context - but LLMs apparently don't?
- magackame 8mo ago
  Don't people do this too all the time?
micw 8mo ago
Maybe that's a bias from training data. I would assume that most documents skip the "clarifying the question/scope" part of reasoning. Imagine a scientific text or even a book. Most will start with a clear context/scope. Either with a thesis or a well defined question or (in case of a book) with a story. Texts that start with a question that first needs to be refined are probably rare.

red75prime 8mo ago

An intelligent mind should keep a balance. U: The car wash is 50m away. Should I walk or should I drive? A: Do you need to wash your car? U: Yep. A: Is the car yours? U: Yep. A: Is your car nearby? U: Yes. A: Do you need to walk to your car? U: No. A: Do you know if the car wash is currently working? U: Yes. A: Does it work? U: YES! A: Do you have a way to pay for the wash? U: YES! A: On you? U: YES! A: Is your clothing appropriate for the occasion? U: Argh...

jiggawatts 8mo ago

This reminds me of a Uni exam that was soooo broken that answering “correctly” entailed guessing how exactly the professor designing the questions misunderstood the topic of his own lectures.

rerdavies 8mo ago
An interesting parallel to that is the "What's the next number in this sequence?" sort of questions. If four numbers are provided, one can calculate the coefficients of a a quartic polynomial, for x values of 0, 1, 2 and 3, and then solve for x=4. Which does indeed provide a defensible "next number". And by similar reasoning, there are an infinite number of answers to this question. Even worse. You could in fact provide any number as an answer, because there is always a quintic polynomial that fits the four initial numbers AND your arbitrary fifth number. So these questions are actually not about what the next number is, but trying to imagine what the person who set the question thought was a "cool" answer, for some curious definition of "cool", for some person who isn't smart enough to realize that the premise on which the question is based is flawed.
1718627440 8mo ago
Are you not allowed to ask the professor questions? We are, and it is not to seldom that the professor then walks to the black board and updates the question.
- jiggawatts 8mo ago
  It was an examination with 300 students in a giant hall, overseen by university staff, not the individual professors. So many people complained that they did eventually fetch him to come and clarify (correct) the questions. I didn’t have the patience to wait for him to turn up, so I simply provided a matrix of solutions for every possible combination of potential original intent… with note next to it saying that anything other than a 100% mark will be met with official complaints about his lack of due diligence.

easyThrowaway 8mo ago

It feels more like a question on english linguistic conventions than logic. If someone asked me the same question and I wanted to give a smartass reply, I'd tell them "You want to wash your car, good to know. Now, about your question, unless you tell me where you wanna go I can't really help you".