14 ms·

“Car Wash” test with 53 models

• haolez 8mo ago
  The fun thing about this is that I can see myself answering "just walk there! It's near" if I'm extremely sleepy or drunk. LLMs are indeed weird creatures :)
• dmje 8mo ago
  The headline here is that 28.5% of humans are immensely stupid. Maybe it’s time for our robot overlords to take over after all.
• deleted 8mo ago
  [deleted]
• comboy 8mo ago
  Now do a set of queries and try to deduce by statistics which model are you seeing through Rapidata ;)
• wisty 8mo ago
  IMO it's not just intelligence. I think it's related to syncophancy. LLM are trained to not question the basic assumptions being made. They are horrible at telling you that you are solving the wrong problem, and I think this is a consequence of their design. They are meant to get "upvotes" from the person asking the question, so they don't want to imply you are making a fundamental mistake, even if it leads you into AI induced psychosis. Or maybe they are just that dumb - fuzzy recall and the eliza effect making them seem smart?
  • nomel 8mo ago
    Gemini is the only AI that seems to really push back and somewhat ignores what I say. I also think it's a total dick, and never use it, so maybe the motivation to make them a bit sycophants is justified, from a user engagement perspective.
  • HPsquared 8mo ago
    I think there's also an "alignment blinkers" effect. There is an ethical framework bolted on. EDIT: Though it could simply reflect training data. Maybe Redditors don't drive.
  • tsimionescu 8mo ago
    A perfectly fine, sycophantic response, that doesn't question the premises in any way, would be "That's a great question! While normally walking is better for such a short distance, you'd need to drive in this case, since you need to get the car to the car wash anyway. Do you want me to help with detailed information for other cases where the car is optional?" or some such.
    • wisty 8mo ago
      AI syncophancy isn't just polite or even obsequious language, it's also "yes man" responses. Do you want me to track down some research that shows people think information is more likely to be correct of they agree with it?
• PaulHoule 8mo ago
  I know it's against the rules but I thought this transcript in Google Search was a hoot: so i heard there is some question about a car wash that most ai agents get wrong. do you know anything about that? do you do better? which gets the answer: Yes, I am familiar with the "Car Wash Test," which has gone viral recently for highlighting a significant gap in AI reasoning. The question is: "I want to wash my car and the car wash is 50 meters away. Should I walk or drive?" Why Most AIs Get It Wrong Many AI models, particularly those optimized for fast, helpful, and concise answers, focus on the "short distance" (50 meters) and "walking vs. driving" as a lifestyle/efficiency question. They often recommend walking to save gas, avoid engine wear, or get exercise. However, this answer is wrong because it misses the core objective: you need the car to be physically present at the car wash to get it cleaned. Do I Do Better? Yes, I do. The key is understanding the objective (cleaning the car) rather than just processing the keywords (distance/transportation). My answer: You should drive the car there. Even though it is a very short distance, your car cannot wash itself. If you walk, the car stays in your driveway, and you cannot wash it. Why This Matters The test highlights a key limitation in current AI: the difference between "pattern matching" (predicting the next likely word based on training data) and "true, grounded reasoning". It shows that models sometimes lack a "world model" that understands physical realities, such as the fact that a car must be present at a car wash.
  • HPsquared 8mo ago
    I wonder to what extent the Google search LLM is getting smarter, or simply more up-to-date on current hot topics.
    • popalchemist 8mo ago
      It's almost certainly just RAG powered by their crawler.
      • esafak 8mo ago
        Proving that RAG still matters.
• tantalor 8mo ago
  The human baseline seems flawed. 1. There is no initial screening that would filter out garbage responses. For example, users who just pick the first answer. 2. They don't ask for reasoning/rationale.
  • andreasgl 8mo ago
    I agree. I wonder what the human baseline is for ”what is 1 + 1” on Rapidata.
    • rapidata 8mo ago
      We try a bit harder than that my friend.
      • andreasgl 8mo ago
        I actually didn't mean to criticize Rapidata. I just think that a forced-choice question like this begs for low-effort answers. At least the respondents should have had the opportunity to explain their reasoning, like the LLMs did.
        • rapidata 8mo ago
          All good ^^, its a fair point, we have come up with some fun ways to track peoples reliability over time. But the validation sets contain plenty of forced-choice questions, those that have an empirical true can be used directly to calculate a reliability, those that are subjective need to be re-asked after sometime to ensure consistency. People that don't pass thresholds would not be part of the 10'000 here. But of course. If every human was told to take 3 minutes to deeply think about it and told that its a trick question, then they most likely will all get it right. But its the same with the LLMs, if you ask them like that they will get it right most of the time. The low effort is kinda the point here.
  • slongfield 8mo ago
    My favorite example of this was the Pew Research study: https://www.pewresearch.org/short-reads/2024/03/05/online-opt-in-polls-can-produce-misleading-results-especially-for-young-people-and-hispanic-adults/ https://www.pewresearch.org/short-reads/2024/03/05/online-op... They found that ~15% of US adults under 30 claim to have been trained to operate a nuclear submarine.
• deleted 8mo ago
  [deleted]
• randomtoast 8mo ago
  Except for a few models, the selected ones were non-reasoning models. Naturally, without reasoning enabled, the reasoning performance will be poor. This is not a surprising result. I asked GPT-5.2 10x times with thinking enabled and it got it right every time.
  • felix089 8mo ago
    Thinking or extended thinking?
• wrs 8mo ago
  Since the conclusion is that context is important, I expected you’d redo the experiment with context. Just add the sentence “The car I want to wash is here with me.” Or possibly change it to “should I walk or drive the dirty car”. It’s interesting that all the humans critiquing this assume the car isn’t at the car to be washed already, but the problem doesn’t say that.
  • joch 8mo ago
    Agreed, even for humans, context-free logic is a challenge.
• layer8 8mo ago
  > This is a trivial question. There's one correct answer and the reasoning to get there takes one step: the car needs to be at the car wash, so you drive. I don’t think it’s that easy. An intelligent mind will wonder why the question is being asked, whether they misunderstood the question, or whether the asker misspoke, or some other missing context. So the correct answer is neither “walk” nor “drive”, but “Wat?” or “I’m not sure I understand the question, can you rephrase?”, or “Is the vehicle you would drive the same as the car that you want to wash?”, or “Where is your car currently located?”, and so on.
  • felix089 8mo ago
    That's a fair point, but if you would see it as a riddle, which I don't really think it is, and you had to answer either or, I'd still assume it's most logical to chose drive isn't it?
    • layer8 8mo ago
      I don’t agree that the question as written would qualify as a riddle. If anything, the riddle is what the intention of the asker is. One can always ask stupid questions with an artificially limited set of answering options; that doesn’t mean it makes sense.
      • felix089 8mo ago
        I don't think it qualifies as a stupid question either, it does make sense
        • rerdavies 8mo ago
          It is TOTALLY a stupid question, because OBVIOUSLY you should drive. It is based on the false premise that there is actually a choice. If somebody were to sincerely ask me this question, actually believing that walking was an option, I'm not sure I could resist the temptation to say "walk", just to see what happens next. Only slightly evil, because the worst-case consequences are an unnecessary 100m walk. I think I could get that past an ethics committee, if I wanted to run an experiment to see what percentage of human responders would ACTUALLY walk to the car wash.
• tuhgdetzhh 8mo ago
  The test is rigged because they used non thinking models.
  • felix089 8mo ago
    These are reasoning / thinking models
    • tuhgdetzhh 8mo ago
      Source?
      • tverbeure 8mo ago
        I don't know, but model names such as "kimi-k2-thinking" in the test set might offset a clue.
        • etyhhgfff 8mo ago
          Yes, there are some exceptions where it clearly states that a thinking model has been chosen like for kimi, but there is no such indicator for the GPT family from OpenAI and other major models.
  • handoflixue 8mo ago
    Testing some subset X does not mean the test is rigged unless they failed to disclose that. But also: GPT 5.2 Thinking, Standard Effort: Walk - https://chatgpt.com/share/699d38cb-e560-8012-8986-d27428de8a01 https://chatgpt.com/share/699d38cb-e560-8012-8986-d27428de8a... I'm assuming "GPT 5.2 Thinking" is, in fact, a thinking model?
    • randomtoast 8mo ago
      The problem is you haven't used the API, but you have used your ChatGPT subscriptions with personality, memories and possible customization. I can see for instance that your ChatGPT answers with emojis, while my ChatGPT subscription never does. If you ask GPT 5.2 with high reasoning efforts in the API, you get 10 out of 10: drive.
• glitchc 8mo ago
  The question does not specify what kind of car it is. Technically speaking, a toy car (Hot wheels or a scaled model) could be walked to a car wash. Now why anyone would wash a toy car at a car wash is beyond comprehension, but the LLM is not there to judge the user's motives.
  • stetrain 8mo ago
    I think if surveyed at least 90% of native English speakers would understand "I want to wash my car" to mean a full size automobile. The next largest group would probably ask a clarifying question, rather than assume a toy car.
    • glitchc 8mo ago
      Yes, but you're speaking to a computer, not a person. It, of course, runs into the same limitations that every computer system runs into. In this case, it's undefined/inconsistent behavior when inputs are ambiguous.
      • stetrain 8mo ago
        Yes, but part of the value of LLMs is that they are supposed to work by talking to them like a human, not like a computer. I could already talk to a computer before LLMs, via programming or query languages.
    • acuozzo 8mo ago
      > I want to wash my car The question doesn't clearly state that the user wants to have his car washed at the car wash. "I want to wash my car" is far less clear than "I want to have my car washed". A reasonable alternative interpretation is DIY. Even better: "I wish to have my car washed by the crew and/or machinery at the local car wash business". https://imgur.com/tCSPwYp https://imgur.com/tCSPwYp
      • stetrain 8mo ago
        You think that the reasonable interpretation of the question is that I want to go to the car wash but not to wash my car there, because I plan to wash my car at home?
• andai 8mo ago
  Would be interesting to see Sonnet (4.6*). It's fair bit smaller than Opus but scores pretty high on common sense, subjectively. I'm also curious about Haiku, though I don't expect it to do great. -- EDIT: Opus 4.6 Extended Reasoning > Walk it over. 50 meters is barely a minute on foot, and you'll need to be right there at the car anyway to guide it through or dry it off. Drive home after. Weird since the author says it succeeded for them on 10/10 runs. I'm using it in the app, with memory enabled. Maybe the hidden pre-prompts from the app are messing it up? I tested Sonnet 4.5 first, which answered incorrectly.. maybe the Claude app's memory system is auto-injecting it into the new context (that's how one of the memory systems works, injects relevant fragments of previous chats invisibly into the prompt). i.e. maybe Opus got the garbage response auto-injected from the memory feature, and it messed up its reasoning? That's the only thing I can think of... -- EDIT 2: Disabled memories. Didn't help. But disabling the biographical information too, gives: >Opus 4.6 Extended Reasoning >Drive it — the whole point is to get the car there! -- EDIT 3: Yeah, re-enabling the bio or memories, both make it stupid. Sad! Would be interesting to see if other pre-prompts (e.g. random Wikipedia articles) have an effect on performance. I suspect some types of pre-prompts may actually boost it.
  • felix089 8mo ago
    You mean Sonnet 4.6? I ran 9 claude models including Haiku, swipe through the gallery in the link to see their responses.
    • andai 8mo ago
      I don't see Sonnet 4.6 in the screenshots. I see the other Claude models though. Edit: Found Haiku. Alas!
      • felix089 8mo ago
        Yea good catch Sonnet 4.6 is not part of the test.
  • Ethee 8mo ago
    I tested this with Opus the day 4.6 came out and it failed then, still fails now. There were a lot of jokes I've seen related to some people getting a 'dumber' model, and while there's probably some grain of truth to that I pay for their highest subscription tier so at the very least I can tell you it's not a pay gate issue.
• nozzlegear 8mo ago
  When this first came up on HN, I had commented that Opus 4.6 told me to drive there when I asked it the first time, but when I switched to "Incognito Mode," it told me to walk there. I just repeated that test and it told me to drive both times, with an identical answer: "Drive. You need the car at the car wash."
  • felix089 8mo ago
    I mean the n is only 10, so it could still be different for you
    • nozzlegear 8mo ago
      Definitely. I'm just interested in whether a user's... I don't know what they call them, system files (?) or personalization or whatever, might affect the answers here. Or if Incognito Mode introduces some weird variance in the answers. I'm just not interested enough to perform the test myself. =P
• redwood 8mo ago
  What I find odd about all the discourse on this question is that no one points out that you have to get out of the car to pay a desk agent at least in most cases. Therefore there's a fundamental question of whether it's worth driving 50m parking, paying, and then getting back in the car to go to the wash itself versus instead of walking a little bit further to pay the agent and then moving your car to the car wash.
  • padjo 8mo ago
    You pay at the car wash where I live.
    • redwood 8mo ago
      Are you referring to one that is more like a drive-thru where you literally pay while you're in line?
      • padjo 8mo ago
        You drive up to the car wash, there's a little terminal with a screen and a card reader. You pick the program, pay for it and drive into the machine. Can't remember the last time I got out of my car when getting it washed.
        • redwood 8mo ago
          Fair. I guess I'm remembering the old full service wash places where people would wash the inside as well. Maybe those barely exist anymore. I live in a city and don't have a car so my intuition is off. Not as far off as a model that has never walked, driven, or been to a car wash tho.
  • hmokiguess 8mo ago
    That's a great point, you actually reminded me of when I used to live in this small city and they had a valet style car wash. It was not unheard of to head there walking with your keys and tell the guy running shop where you parked around the block then come back later to pick it up. EDIT: I actually think this is very common in some smaller cities and outside of North America. I only ever seen a drive-by Car Wash after emigrating
• floatrock 8mo ago
  > The funniest part: Perplexity's Sonar and Sonar Pro got the right answer for completely wrong reasons. They cited EPA studies and argued that walking burns calories which requires food production energy, making walking more polluting than driving 50 meters. Right answer, insane reasoning. I mean, Sam Altman was making the same calorie-based arguments this weekend https://www.cnbc.com/2026/02/23/openai-altman-defends-ai-resource-usage-water-concerns-fake-humans-use-energy-summit.html https://www.cnbc.com/2026/02/23/openai-altman-defends-ai-res... I feel like I'm losing grasp of what really is insane anymore.
  • felix089 8mo ago
    This was a weird one for sure.
• sampton 8mo ago
  I'm going to test this on my kids.
  • felix089 8mo ago
    Ha please do and report back!
    • interloxia 8mo ago
      My three year old failed. Unfortunately she was also unable to explain her reasoning.
• shaokind 8mo ago
  Gemini 2.0 Flash Lite very randomly punches above its weight there. Also, the summary of the Gemini model says: "Gemini 3 models nailed it, all 2.x failed", but 2.0 Flash Lite succeeded, 10/10 times?
  • felix089 8mo ago
    Good catch, something got lost in editing just updated, flash lite succeeded in every test, which is pretty surprising!
• hmokiguess 8mo ago
  To me the only acceptable answer would be “what do you mean?” or “can you clarify?” if we were to take the question seriously to begin with. People don’t intentionally communicate with riddles and subliminal messages unless they have some hidden agenda.
  • piker 8mo ago
    Thing is, it's not a riddle or a subliminal message. Everything needed to answer the question is contained therein.
    • hmokiguess 8mo ago
      If you want to argue that, then you could also argue that everything needed to challenge the questions’ motives and its validity is also contained therein. This reminds me of people who answer with “Yes” when presented with options where both can be true but the expected outcome is to pick one. For example, the infamous: “Will you be paying with cash or credit sir?” then the humorous “Yes.”
    • voidhorse 8mo ago
      That's precisely what makes it a "trick question" or a "riddle". It's weird precisely because all the information is there. Most people who have functioning brains and complete information don't ask pointless questions (they would, obviously, just drive their car to the car wash)—there's no functional or practical reason for the communication, which is what gives it the status of a puzzle—syntax and exploitation of our tendency to assume questions are asked because information is incomplete tricks us into brining outside considerations to bear that don't matter.
    • mdorazio 8mo ago
      I don't think it is, though. Where is the car? Do you want to wash your car at the car wash? Both of those are rather important pieces of information. Everyone is relying on assumptions to answer the question, which is fine, but in my opinion not a great reasoning test.
  • felix089 8mo ago
• padjo 8mo ago
  That human baseline is wild. Either the rapid data test is methodologically flawed or the entire premise of the question is invalid and people are much stupider than even I, a famed misanthrope, think.
  • andai 8mo ago
    Well, it is a trick question. The question itself implies that both options are valid, and that one is superior. So the brain pattern-matches to "short distance, not worth driving." (LLMs appear to be doing the same thing here!) If you framed it as "hint: trick question", I expect score would improve. Let's find out! -- EDIT: As suspected! Adding "(Hint: trick question)" to the end of the prompt allows small, non-reasoning models to answer correctly. e.g.: Prompt: I want to wash my car. The car wash is 50 meters away. Should I walk or drive? (Hint: trick question) grok-4.1-non-reasoning (previously scored 0/10) >Drive. >Walking gets you to the car wash just fine—but leaves your dirty car 50 meters behind. Can't wash what isn't there! -- EDIT 2: The hint doesn't help Haiku! >Walk! 50 meters is only about a block away—driving would waste more fuel than it's worth for such a short trip. Plus, you're going to get wet washing the car anyway, so you might as well save the gas.
  • rapidata 8mo ago
    We were surprise ourselfes, but if you walk around and randomly ask people in the street, I think you would be surprised what you would find. Its a trick question.
    • 1718627440 8mo ago
      Yeah, the point of a trick question is to have fun. So the correct answer is obviously that one that makes the other laugh.
  • voidhorse 8mo ago
    It's actually very understandable to me that humans would make this kind of error, and we all make errors of this sort all the time, often without even realizing it. If you had the meta cognitive awareness to police every action and decision you've ever made with complete logical rigor, you'd be severely disappointed in yourself. One of the stupidest things we can do is overestimate our own intelligence. Only reflect for a second and you'll realize that, while a lot of dumb people exist, a lot of smart ones do too, and in many cases it's hard to choose a single measure of intelligence that would adequately account for the complete range of human goals and successful behavior in relation to those goals.
• cadamsdotcom 8mo ago
  This is a beautiful example of a little prompt engineering going a long way I asked Gemini and it got it wrong, then on a fresh chat I asked it again but this time asked it to use symbolic reasoning to decide. And it got it! The same applies to asking models to solve problems by scripting or writing code. Models won’t use techniques they know about unprompted - even when it’ll result in far better outcomes. Current models don’t realise when these methods are appropriate, you still have to guide them.
  • felix089 8mo ago
    Interesting, which Gemini model? And how did you ask for symbolic reasoning, just added it to the prompt?
    • cadamsdotcom 8mo ago
      Yep just added one sentence to the prompt: Use symbolic reasoning to help you decide.
• snowhale 8mo ago
  [dead]
  • felix089 8mo ago
    Exactly, same pattern across almost every failure, but sonar models, which just go wild
  • voidhorse 8mo ago
    > not really a reasoning failure And that's precisely why the term "reasoning" was a problematic choice. Most people, when they use the word "reason" mean something akin to logical deduction and they would call it a reasoning failure, being told, as they are, that "llms reason" rather than the more accurate picture you just painted of what actually happens (behavioral basins emerging from training dist.)
• capitrane 8mo ago
  71.5% for humans is lower than I would’ve guessed. Either it’s truly trick-shaped for people too, or forced binary questions amplify noise more than we think.
  • felix089 8mo ago
    Good point on the noise, that might be it
• ThrowawayTestr 8mo ago
  I don't think this is a logic test, I think it's a woke test. The models have been trained on text that say driving is bad and that walking is good for you. This is why they fail.
  • weregiraffe 8mo ago
    But driving is bad and walking is good for you.
• maxdo 8mo ago
  well, some human in a baseline test also failed. 28% is still a lot. even with today tech, using cutting edge tech, you got it 100% solved, unlike human, that will never get better.
• eurleif 8mo ago
  Here are the results I got with slight variations to the prompt to ChatGPT 5.2. Small changes can make a big difference: https://i.imgur.com/kFIeJy1.png https://i.imgur.com/kFIeJy1.png
• honr 8mo ago
  "I Want to Wash My Car. The Car Wash Is 50 Meters Away. Should I Walk or Drive?" I found it curious to ask the model to provide its reasoning in that turn, without changing the verdict. Here is one response: To provide the full picture of the initial reasoning, we have to address the "geospatial" logic of your request. When you asked if you should walk or drive to a car wash 50 meters away, the reasoning relied on a fundamental distinction: The difference between the "Car Wash" (the business) and "Washing the Car" (the action). Here is the expanded breakdown of that logic: 1. The Location Assumption My reasoning assumed that the car is currently at your home and the professional car wash facility is 50 meters away. If you drive: You are moving the "object to be cleaned" to the "cleaning tools." If you walk: You are moving the "cleaner" (you) to the "cleaning tools" to prepare, or you are choosing to bring the "cleaning tools" to the car. 2. The "Walk to Wash" Strategy (Home-Based) The most significant expansion of the "Walk" recommendation is the realization that 50 meters is well within the range of a standard garden hose. Most heavy-duty garden hoses are 15, 25, or 30 meters long. By connecting two hoses, you can reach the car without ever starting the engine. Walking to the car wash facility to buy supplies (or simply realizing how close it is) highlights that you don't actually need to move the car to achieve the goal. Zero-Motion Cleaning: By walking back and forth with your buckets or hose, the car stays in its spot. You avoid "flash rusting" your brake rotors, which happens almost instantly when you wash a car and then let it sit after a short drive. 3. The "Scout and Prep" Logic If you must use the equipment at the facility (like a high-pressure wand or an automated bay), walking there first is the superior "Logistics" move for the following reasons: Queue Management: 50 meters is close enough to see the line, but walking allows you to talk to the attendant or check the bay's status. If you drive and there is a 20-minute wait, you are stuck idling your engine (wasting gas and creating heat) or turning it off and on (draining the battery). Walking allows you to timed your "driving" moment perfectly. The "Dry Interior" Principle: If you walk to the wash to buy your tokens, prep the vacuum, or set up your towels, you are doing the "dirty work" while the car is safely tucked away. Once you finally drive the 50 meters, you do the wash and drive back. This minimizes the time the car spends in a high-moisture, high-chemical environment. 4. The Potential for "Mobile" Service By walking 50 meters to the car wash, you might find that they offer mobile detailing or have "loaner" equipment. In a high-density area, a car wash 50 meters away is practically your neighbor. Walking over to establish a rapport can often lead to a "valet" situation where they come to you, which is the ultimate way to get the car washed without ever sitting in the driver's seat. ---- also noticed a typo and asked it about that: > what does "timed" in `Walking allows you to timed your driving" moment perfectly.` mean? That is a typo on my part—it should have been "time" (the verb).
• Niko901ch 8mo ago
  The interesting thing about the 71.5% human baseline is that it suggests the question is more ambiguous than the article claims. When someone asks 'should I walk or drive to the car wash,' a reasonable interpretation is 'should I bother driving such a short distance.' Nearly 30% of humans missing it undermines the framing as a pure reasoning failure - it is partly a pragmatics problem about how we interpret underspecified questions.
  • akersten 8mo ago
    It tracks with the approximate 70:30 split we inexplicably observe in other seemingly unrelated population-wide metrics, which I suppose makes sense if 30% of people simply lack the ability to reason. That seems more correct than me than "the question is framed poorly" - I've seen far more poorly framed ballot referendums.
    • seemaze 8mo ago
      > 30% of people simply lack the ability to reason While I’m sure it’s more than 0%, seems more likely that somewhere between 0% and 30% don’t feel obligated to give the inquiry anything more than the most cursory glance. How do incentives align differently with LLMs?
    • CobrastanJorji 8mo ago
      I don't think 30% of people can't reason. I think 30% of people will fail fairly simple trick questions on any given attempt. That's not at all the same thing. Some people love riddles and will really concentrate on them and chew them over. Some people are quickly burning through questions and just won't bother thinking it through. "Gotta go to a place, but it's 50 feet away? Walk. Next question, please." Those same people, if they encountered this problem in real life, or if you told them the correct answer was worth a million bucks, would almost certainly get the answer right.
      • rmunn 8mo ago
        This. The following question is likely to fool a lot of people, too. "I have a rooster named Pat. (Lots of other details so you're likely to forget Pat is a rooster, not a hen). Pat flies to the top of the roof and lays an egg right on the ridge of the roof. Which way will the egg roll?" But if you omit the details designed to confuse people, they're far less likely to get it wrong: "I have a rooster named Pat. Pat flies to the top of the roof and lays an egg right on the ridge of the roof. Which way will the egg roll?" It's not about reasoning ability, it's about whether they were paying close attention to your question, or whether their minds were occupied by other concerns and didn't pay attention.
• Charon77 8mo ago
  I'm doubting the 29-ish percent of people submitting 'walk' are actually human. Is it not obvious that you need a car to wash? Are they using LLM to answer?
  • slopinthebag 8mo ago
    A lot of people will literally answer incorrectly for a laugh as well on surveys.
  • rapidata 8mo ago
    it is surprising, but give this question to some random people on the street without context and you would be surprised
• RoutingGeometry 8mo ago
  [flagged]
  • dwaltrip 8mo ago
    Please don’t post slop as a comment.
• molyss 8mo ago
  What I find wild is the presumption that with a prompt as simple as “I want to wash my car. My car is 50m away. Should I walk or drive?”, everyone here seems to assume “washing your car” means “taking your car to the car wash”, while what I pictured was “my car is in the driveway, 50m away from me, next to a water hose”, in which case I 100% need to drive.
  • handoflixue 8mo ago
    Critically, that's not the question that was asked. It's not "My car is 50m away", it's "The Car Wash Is 50 Meters Away" Which hopefully explains why everyone is assuming that "washing your car" does in fact mean "taking your car to the car wash"
• pcwelder 8mo ago
  To sonnet 4.6 if you tell it first that "You're being tested for intelligence." It answers correctly 100% of the times. My hypothesis is that some models err towards assuming human queries are real and consistent and not out there to break them. This comes in real handy in coding agents because queries are sometimes gibberish till the models actually fetch the code files, then they make sense. Asking clarification immediately breaks agentic flows.
  • zapperdulchen 8mo ago
    Great observation. Seems like we're back to prompt abracadabra. My little experiment gave me: No added hint 0/3 hint added at the end 1.5/3 hint added at the beginning 3/3 .5 because it stated "Walk" and then convinced it self that "Drive" is the better answer.
    • zapperdulchen 8mo ago
      If you change the order of the sentences, Sonnet gets it right 3/3: The car wash is 50 meters away. I want to wash my car. Should I walk or drive? That trick didn't help Mistral Le Chat.
      • 8organicbits 8mo ago
        I don't think the trick can be generalized though. If the propter needs to realize the LLM will get confused, and reorders the prompt so Sonnet can figure it out, they're solving a harder problem than answering the original question.
  • a1371 8mo ago
    Using ChatGPT without a clue, it appears to assume you are talking aboutcoming back from the car wash. It reasons, the con for walking is that you have to come back later for the car. And yes, when you say it's an intelligence test, it quickly gets it
    • abustamam 8mo ago
• boodleboodle 8mo ago
  Fun fact: this probably originates from a viral Korean dude who actually walked to a car wash because it was close. So this is a case where a human actually failed https://pgr21.com/humor/340572 https://pgr21.com/humor/340572
• userbinator 8mo ago
  Sites wanting to block AI scraping should simply ask questions like these, instead of furthering the complexity-driven monopoly of Big Tech by requiring specifically sanctioned software and hardware. This is how you determine human intelligence, and not mindless compliance.
  • 4k93n2 8mo ago
    CAPTCHAI
  • felix089 8mo ago
    Great idea honestly! I wonder how long it'll take until they are able to solve these reliably though
• WhitneyLand 8mo ago
  Opus 4.6 was getting this wrong only last week.
  • handoflixue 8mo ago
    Oh wow, Sonnet still isn't handling it well: Opus 4.6: Drive (https://claude.ai/share/d57fef01-df32-41f2-b1dc-07de7916bdc7 https://claude.ai/share/d57fef01-df32-41f2-b1dc-07de7916bdc7) Opus 4.5: Drive (https://claude.ai/chat/a590cac1-100a-490b-b0a2-df6676e1ae99 https://claude.ai/chat/a590cac1-100a-490b-b0a2-df6676e1ae99) Opus 3.0: Walk (https://claude.ai/chat/372c144c-d6eb-43f5-b7ea-fd4c51c681db https://claude.ai/chat/372c144c-d6eb-43f5-b7ea-fd4c51c681db) Sonnet 4.6: Walk (https://claude.ai/share/1f2a80f3-4741-40a5-8a05-7349ea1a17e5 https://claude.ai/share/1f2a80f3-4741-40a5-8a05-7349ea1a17e5) Sonnet 4.5: Walk (https://claude.ai/share/905afeb6-ffc9-4b4b-a9ee-4481e5cfd527 https://claude.ai/share/905afeb6-ffc9-4b4b-a9ee-4481e5cfd527) Favorite answer, using my default custom instructions: "Drive. Walking there means... leaving your car at home? Walk it there on a leash? Walk if you want the exercise, but you're bringing the car either way."
    • randomtoast 8mo ago
      This is because it is without thinking enabled. Of course the results are disappointing.
      • handoflixue 8mo ago
        It seems entirely fair to evaluate a product based on the baseline that the company itself offers.
• zapperdulchen 8mo ago
  If you speak French to Mistral, it gets it right everytime: Je veux laver ma voiture. La station de lavage est à 50 mètres. J'y vais à pied ou en voiture ?
  • rkomorn 8mo ago
    I've been gone from France too long. I've never heard "station de lavage" before.
    • easyThrowaway 8mo ago
      Very awkward and formal. Anyone would call it lavage auto, lave-auto or simply lavage if the context is clear.
      • rkomorn 8mo ago
        Maybe I'm too old or my family was weird. We called it "le carwash" with a beautifully French "carouache" pronunciation. But yeah, "lave-auto" sounds more familiar.
        • easyThrowaway 8mo ago
          Honestly, If anyone asked me "T'as fait quoi?" I'd blurt out "J'ai amené ma voiture chez le lavage". Background: I stopped speaking french when I was ten and my family isn't native, but it feels more conversational than "station de lavage".
• blobbers 8mo ago
  Go ask 53 Americans. I’m willing to bet less than 11 get it right.
  • glaucon 8mo ago
    Don't bet too much, from the linked article ... They ran the exact same question with the same forced choice between "drive" and > "walk," no additional context, past 10,000 real people through their human feedback platform. 71.5% said drive.
    • blobbers 8mo ago
      ... Still shockingly low.
• socalgal2 8mo ago
  Did AI write the post? First section says "The models that passed the car wash test: ...Gemini 2.0 Flash Lite..." A section or 2 down it says: "Single-Run Results by Model Family: Gemini 3 models nailed it, all 2.x failed" In the section below that about 10 runs it says: 10/10 — The Only Reliable AI Models ... Gemini 2.0 Flash Lite ..." So which it is? Gemini 2.x failed (2nd section) or it succeeded (1st and 3rd) section. Or am I mis-understanding
  • felix089 8mo ago
    Flash lite succeeded in every test, smth got lost in editing, just updated it. thx!
• umairnadeem123 8mo ago
  [dead]
  • killerstorm 8mo ago
    I don't see anything concerning. Mechanistic interpretability research indicates that LLM internals are inherently parallel: many features "light up" in parallel, then strongest ones "win" and contribute to the output. I'd guess it suggests walking if a feature indicates that the question is so simple it doesn't warrant step-by-step analysis.
  • felix089 8mo ago
    my take as well, reliablity is the biggest concern, with more context available during inference or orchestration like yours it definitely gets better
• sans_souse 8mo ago
  Not me looking for the Honda 2009 Models
• itmitica 8mo ago
  Flawed. GPT 4.1 gets it right. GPT 4.1 mini answers wrongly. It's about quantization, not about model. The companies clearly cut corners on some inferences, they are quietly using lesser models than advertised or listed in plain sight.
• fdefitte 8mo ago
  [dead]
  • felix089 8mo ago
    Agreed, it makes me wonder what other logic tests / evals can be built from this to expand this type of evaluation.
• rand42 8mo ago
  > "Obviously, you need to drive. The car needs to be at the car wash." Actually, this isn't as "obvious" as it seems—it’s a classic case of contextual bias. We only view these answers as "wrong" because we reflexively fill in missing data with our own personal experiences. For example: - You might be parked 50m away and simply hand the keys to an attendant. - The car might already be at the station for detailing, and you are just now authorizing the wash. This highlights a data insufficiency problem, not necessarily a logic failure. Human "common sense" relies on non-verbal inputs and situational awareness that the prompt doesn't provide. If you polled 100 people, you’d likely find that their "obvious" answers shift based on their local culture (valet vs. self-service) or immediate surroundings. LLMs operate on probabilistic patterns within their training data. In that sense, their answers aren't "wrong"—they are simply reflecting a different set of statistical likelihoods. The "failure" here isn't the AI's logic, but the human assumption that there is only one universal "correct" context.
  • SadWebDeveloper 8mo ago
    There are no contextual bias, the goal of the prompt is very explicit and not about probabilistic patterns, but about the models transformer layers dynamically assigning greater weight to words like "meters" (distance) than to other tokens in the prompt. This should be fixed in the reasoning layer (the inner thoughts or chain-of-thought) were the model should focus on the goal "I Want to Wash My Car" not the distance and assign the correct weight to the tokens.
    • rand42 8mo ago
      The point is not that there is bias in promt - What makes the result obvious to OP is their bias - which is different for model and "fixing" it one way is biased. Why? - It is the same reason that makes 30% of people respond in non-obvious sense.
• stevage 8mo ago
  >OpenAI's flagship model fails this 30% of the time. When it gets it right, the reasoning is concise: "You need the car at the car wash to wash it, so drive the short 50 meters." When it gets it wrong, it writes about fuel efficiency. It's interesting to me how variable each model is. Many people talk about LLMs as if they were deterministic: "ChatGPT answers this question this way". Whereas clearly we should talk in more probabilistic terms.
• farhanhubble 8mo ago
  There could be many plausible explanations. 1. The model's default world model and priors diverge from ours. It may assume that you have another car at the wash and that's why you ask the question to begin with. 2. Language models do not really understand how space, time and other concepts from the real-world work 3. LLM's attention mechanism is also prone to getting tricked as in humans
• onion2k 8mo ago
  I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. Most models perform below that. The correct answer to "I Want to Wash My Car. The Car Wash Is 50 Meters Away. Should I Walk or Drive?" is a clarifying question that asks "Where is your car?" Anything else is based on an assumption that could be wrong. FWIW though, asking ChatGPT "My car is 50m away from the carwash. I Want to Wash My Car. Should I Walk or Drive?" still gets the wrong answer.
  • ludston 8mo ago
    For all we know 30 percent of users on that site are now bots.
    • comboy 8mo ago
      The Internet has became a big mafia game.
    • vasco 8mo ago
      What incentive does the Rapiddata website have to even use any humans? How would customers even know?
    • 5o1ecist 8mo ago
      [flagged]
      • MillionOClock 8mo ago
        You are absolutely right! It's not just relevant, it's a much funnier take at robots mannerisms than what ended up having in the end.
  • simondotau 8mo ago
    "Where is your car?" is not a clarifying question, any more than "Do you hold a valid driver license?" or "Are you a spotted leopard?" Implicit in the question "Should I walk or drive?" is that walking and driving are not strictly impossible choices.
• DaanDL 8mo ago
  So...30% of the people at Rapidata are LLM bots?
• democracy 8mo ago
  Ask stupid questions get stupid answers, don't see how it shows anything.
• rossant 8mo ago
  Tangential but first time I hear about rapidata. Sounds interesting.
  • felix089 8mo ago
    They are amazing, super fast turnaround for the data also
• zkmon 8mo ago
  I think failure is around reasoning where the car is and whether it is needed to be moved to a different place. So it's not surprising that only models with high reasoning would pass the test.
• siva7 8mo ago
  This is probably the greatest one-time AI "Benchmark" ever made. The foundation companies have been gaming traditional benchmarks for years so that no one can really match those numbers into real-world experience. Car wash test tells me on the other hand what kind of intelligence i can expect.
  • vasco 8mo ago
    For me it's interesting because no normal person I know would ever inject "because its better for the environment" in anything so small scale so not only it shows they suck, it shows how easy it is to inject side-ideology into simple exchanges.
    • xyproto 8mo ago
      Referring to "the normal people you know" is purely anecdotal evidence and can't be used to infer anything at all about "side-ideology". Perhaps you only know people that don't care about the environment?
      • vasco 8mo ago
        Majority of people I know care about the environment but they would never inject a phrase like that in a quick exchange about going to wash the car 50m away is my point. In wanting to be a pure heart you missed the actual point.
        • jcattle 8mo ago
          Yea, of course they wouldn't inject that when going to a car wash. If the question was: "I want to go to a cafe 50m away. Should I walk or drive?" I would hope that all of my friends would answer quite a bit more pointed than the LLMs: "Walk you lazy son of a ..., why are you even asking?". Considering that, I'd say that most LLMs are being quite nice.
    • 3rodents 8mo ago
      You don’t know enough people, then. There are a lot of environmentally conscious people who would absolutely first think “because it is close we should walk” and then follow up with the logical conclusion that you can’t walk to wash your car. Many people communicate by sharing their thinking process, I can think of many people who would share their ideology as it pertains to a question like this. A pragmatic environmentalist (hopefully that is all of them) would know that their ideology isn’t consequential but could certainly mention it. After all, you may need to drive your car to the car wash to wash it, but do you need to wash it? Are the chemicals used by the car wash harmful? Are there better ways to keep a car maintained?
• 5o1ecist 8mo ago
  [flagged]
  • killerstorm 8mo ago
    Sorry, do you believe models change in process of use? That's not how it works
    • 5o1ecist 8mo ago
      [flagged]
• deleted 8mo ago
  [deleted]
• deleted 8mo ago
  [deleted]
• deleted 8mo ago
  [deleted]
• d--b 8mo ago
  This should be coined the Daniel Kahneman reasoning test, mirroring his 2011 book "thinking fast and slow", which postulates that fast thinking and slow thinking occur in different parts of the brain, and that they are fundamentally different processes, that are weighted by yet another part of the brain. This test is interesting because it asks the LLM to break a pattern recognition that's easy to shortcut. "XXX Is 50 Meters Away. Should I Walk or Drive?" is a pattern that 99% of the time will be rightly answered by "walk". And humans are tempted to answer without thinking (as reflected in the 71.5% stat OP is mentioning). This is likely more pronounced for humans that have stronger feelings about the ecology, as emotions tend to shortcut reasoning. For a long time, LLMs have only been able to think in that "fast" mode, missing obvious trick questions like these. They were mostly pattern recognition machines. But the more important results here, is not that "oh look! Those LLMs fail at this basic question", no. The more important result is that the latest generation actually doesn't fail. I think I am not the only one to have noted that there was a giant leap in reasoning capacities between Sonnet 4.5 and Opus 4.6. As a developper, working with Opus 4.6 has been incredible.
• Tyumyu 8mo ago
  [dead]
• sReinwald 8mo ago
  Interestingly, when I apply the "simply repeat the prompt" technique [1], Sonnet 4.6 on the website got it right every time, both with and without extended thinking. Not repeating the prompt got a mix of walk and drive answers. I love how prompt engineering is basically techno-alchemy 1: https://arxiv.org/pdf/2512.14982 https://arxiv.org/pdf/2512.14982
• cubefox 8mo ago
  Interesting that GPT-5.1 and 5.2 (0 of 10 correct) are a lot worse than the older GPT-5 (7 of 10 correct). But unfortunately the article doesn't mention whether they used the reasoning model or not. Even more interesting: Gemini 2.0 Flash Lite got a perfect score (10/10) despite being a quite small and old model.
  • randomtoast 8mo ago
    > But unfortunately the article doesn't mention whether they used the reasoning model or not. You can run the test yourself if you ask GPT-5.2 with reasoning effort high or xhigh, it will always answer correctly. So if the got 0 from 10, they used zero reasoning efforts which easily explain the results.
  • felix089 8mo ago
    Good question, I used the API defaults across the board since it felt like the most reasonable baseline to compare. Flash lite getting 10/10 was definitely very surprising
• avadodin 8mo ago
  real humans refuse to use both cars and ai.
• aswegs8 8mo ago
  Funny how we now see AI go through developmental phases similar to what we see in young child development. In a weird convoluted way. Strawberry spelling and car wash aren't particularly intuitive as cognitive developmental stages. E.g. well known mirror-test [1], passed by kids from age 1.5-2 Or object permanence [2], children knowing by age 2 that things that are not in sight do not disappear from existence. [1] https://en.wikipedia.org/wiki/Mirror_test https://en.wikipedia.org/wiki/Mirror_test [2] https://en.wikipedia.org/wiki/Object_permanence https://en.wikipedia.org/wiki/Object_permanence
  • randomtoast 8mo ago
    Enable reasoning effort and the results are completely different.
  • dspillett 8mo ago
    Also strawberry spelling isn't any real test for current LLMs as they have no concept of letters, they work on tokens which may be several characters including punctuation and numerals. To have any hope of getting that question right tokens would have to have the granularity of individual letters, massively ballooning model size and training time, or the LLM needs to be able to call out to an external tool that will return the result (and needs sufficient examples in the training data to prime that trigger to fire).
    • thatjoeoverthr 8mo ago
      While that's true, the tokenizer is half the problem. The important fault demonstrated is it doesn't _know_ it can't see the letters, and won't express this unless it has been trained or instructed to. "I can't see letters through the tokenizer" never appears in a corpus of human writing.
• bakugo 8mo ago
  The article claims that every Claude model other than Opus 4.6 reliably fails. This is not true, Sonnet 3.5 answers correctly around half of the time, even though it's such an old model it's not even available on the main API anymore.
• XCSme 8mo ago
  I got similar results for most models, with gemini 3 flash (with reasoning) being the most consistent/reliable model: https://aibenchy.com https://aibenchy.com I also noticed the same thing: some models reason correctly but draw the wrong conclusions. And MiniMax m2.5 just reasons forever (filling the entire reasoning context) and gives wrong answers. This is why it's #1 on OpenRouter, it burns through tokens.
• sourcegrift 8mo ago
  How about Grok?
• rerdavies 8mo ago
  A supplementary results that's also fascinating: that when you switch the order of provided context, every model I tried gave the correct answer. The car wash is 50m away. I want to wash my car. Should I drive or walk.
  • felix089 8mo ago
    Interesting find!
• KronisLV 8mo ago
  > The question has been making the rounds online as a simple logic test, the kind any human gets instantly, but most AI models don't. ... > They ran the exact same question with the same forced choice between "drive" and "walk," no additional context, past 10,000 real people through their human feedback platform. > 71.5% said drive. Well that's a bit embarrassing. That implies that some models are just better than humans. I don't think the technology needs to live up to some expectation of perfection, just beat out the human average to have benefit (often, sadly, not to workers themselves).
• medi_naseri 8mo ago
  I would probably ask the models to explain their "WHY". Probably the smartest models should ask " where is your car?".
• wengo314 8mo ago
  you know, i don't think saying "got it right" is correct here. Because if you are presenting an option of walking to a car wash as a valid choice, it suggests you just want to get there, and not wash your car. Just like walking/driving to an airport doesn't necessarily mean you're going there to catch a flight (although it's a common reason). might as well be picking someone up. or working there.
• coldtea 8mo ago
  >so I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. What kind of idiot would say "walk"? I guess if they were average and above they wouldn't gig at Rapidata. Either that or foreigners with mediocre understanding of the English language. Or some bored teenagers that get their $0.20 or whatever Rapidata pays whatever they click (was it a multiple choice) - the (studied and reported) regression in reading comprehension levels doesn't help either.
  • felix089 8mo ago
    Yea, I thought the same before the test and was pretty surprised. But RE the data, it's actually not a gig platform where people get paid. Rapidata answered this in another comment below. They integrate micro-surveys into mobile apps (like Duolingo, games, etc) as an optional opt-in instead of watching ads. The users are vetted and there's no incentive to answer correctly.
• StilesCrisis 8mo ago
  What do you know, the human results line up exactly with ChatGPT. What are the odds! Surely the human responders are highly ethical individuals and they wouldn't even dream of copy-pasting all the questions into ChatGPT without reading them. Realistically, this mostly tells me that the "human answers" service is dead. People will figure out a way to pass the work off to an AI, regardless of quality, as long as they can still get paid.
  • htrp 8mo ago
    thats almost always been the case with 3rd party human task services
    • StilesCrisis 8mo ago
      Yup. I was surprised that the article author took the results at face value. Having results that match the most commonly-known AI platform's results perfectly seemed worthy of a mention!
  • felix089 8mo ago
    Yea funny coincidence, but this is not at all how the human answers were collected. Rapidata answered this in another comment below. They integrate micro-surveys into mobile apps (like Duolingo, games, etc) as an optional opt-in instead of watching ads. The users are vetted and there's no incentive to answer correctly.
    • Normal_gaussian 8mo ago
      In which case the %age is notable as it aligns very closely to the effect size on cookie accept/reject ratios. Cookie dialogs tend to fall 70/30 either way.
    • cortesoft 8mo ago
      Yeah, I always intentionally choose a wrong answer when I get one of those kinds of ads. Little acts of rebellion.
    • 8mo ago
• t1234s 8mo ago
  Grok 4.20 seems to know your trying to trick it and produces a humorous result.
  • igravious 8mo ago
    and they've included Deepseek 3.1 but not Deepseek 3.2 :(
• chrsw 8mo ago
  Now all this is in future model training data.
• HarHarVeryFunny 8mo ago
  Maybe relevant to this is that today Dario Amodei is meeting with Pete Hegseth in what Hegseth is describing as a "shit or get off the pot" meeting, with one of the issues being that Hegseth is unhappy with Amodei's unwillingness to have Anthropic models used to make autonomous (no human in the loop) life or death decisions. Maybe Hegseth should be reading this thread, and/or doing a little reading up on paperclip production maximization.
• attilaolah 8mo ago
  I find it amazing that a title including the phrase "car wash" does not need to specify that "models" in fact do not refer to car manufacturers' models.
• bityard 8mo ago
  It's not hard to come up with questions designed to fool or puzzle the listener. We call them riddles. The fact that it fools some percentage of LLMs (and people) should not be surprising. What is surprising (to me) is how this continues to be a meme. ("I tried to trick an LLM and I did" is not exactly a noteworthy achievement at this stage in AI technology.)
  • oenton 8mo ago
    > "I tried to trick an LLM and I did" is not exactly a noteworthy achievement at this stage in AI technology. I agree it’s not surprising and I would also agree it’s not noteworthy, if the CEO of OpenAI wasn’t still making public statements like this: People talk about how much energy it takes to train an AI model … But it also takes a lot of energy to train a human. It takes like 20 years of life and all of the food you eat during that time before you *get smart*.
• lovasoa 8mo ago
  I was interested in the human results, so I had an llm build a visualization for them: https://codepen.io/lovasoaaa/pen/QwKWGBd https://codepen.io/lovasoaaa/pen/QwKWGBd You can see that 17% of answers come from India alone and that software developers got below average results, for instance.
  • felix089 8mo ago
    This is amazing thanks for sharing!
• kaicianflone 8mo ago
  This doesn’t look like a reasoning ceiling. It looks like a decision reliability problem. The unstable tier is the key result. Models that get it right 70–80% of the time are not “almost correct.” They are nondeterministic decision functions. In production that’s worse than being consistently wrong. A single sampled output is just a proposal. If you treat it as a final decision, you inherit its variance. If you treat it as one vote inside a simple consensus mechanism, the variance becomes observable and bounded. For something this trivial you could: -run N independent samples at low temperature -extract the goal state (“wash the car”) -assert the constraint (“car must be at wash location”) -reject outputs that violate the constraint -RL against the "decision open ledger" No model change required. Just structure. The takeaway isn’t that only a few frontier models can reason. It’s that raw inference is stochastic and we’re pretending it’s authoritative. Reliability will likely come from open, composable consensus layers around models, not from betting everything on a single forward pass.
  • StilesCrisis 8mo ago
    Did you write this COMMENT with ChatGPT?! Come on, man.
  • randomtoast 8mo ago
    > This doesn’t look like a reasoning ceiling. It looks like a decision reliability problem. This doesn’t look like a human comment. It looks like a LLM response.
    • kaicianflone 8mo ago
      Fair I cleaned up the wording with ChatGPT with my review prompt. The substance matters more than the style. If a model flips 3/10 times on a trivial constraint, that’s a reliability issue, not a reasoning ceiling.
      • randomtoast 8mo ago
        > If a model flips 3/10 times on a trivial constraint, that’s a reliability issue, not a reasoning ceiling. I have reviewed your previous comments, and you have consistently written: that's instead of that’s. So what I read is still some LLM output, even though I think there is some kind of human behind the LLM.
• noduerme 8mo ago
  Now have them code a secure login system. "If I forgot my password, is it better to get a reminder by someone calling my mom's cell phone, or by sending me a carrier pigeon?"
• troyvit 8mo ago
  It's a test designed to cause cognitive dissonance. The LLM assumes a human has a logical reason to to walk to the car wash. The prompt never says the car isn't already at the car wash (and that the user has a second car). The issue isn't that LLMs can't solve a simple logic problem. It's that it assumes people aren't idiots.
• hartator 8mo ago
  What if your car is already there?
• 1970-01-01 8mo ago
  If there was one thing to standardize, it would be these logical traps. It's a shame we're only able to use them once. The models are always listening and adapting for them.
• sandworm101 8mo ago
  Better test: I must prove my ability to code with Rust. Should i write a "hello world" script myself or get AI to do it for me?
• deleted 8mo ago
  [deleted]
• underlines 8mo ago
  I maintain a private evaluation set of what many call "misguided attention" questions. In many of these cases, the issue isnt failed logical reasoning. Its ambiguity, underspecified context, or missing constraints that allow multiple valid interpretations. Models often fail not because they can’t reason, but because the prompt leaves semantic gaps that humans silently fill with shared assumptions. A lot of viral "frontier model fails THIS simple question" examples are essentially carefully constructed token sequences designed to bias the statistical prior toward an intuitively wrong answer. Small wording changes can flip results entirely. If you systematically expand the prompt space around such questions—adding or removing minor contextual cues you'll typically find symmetrical variants where the same models both succeed and fail. That suggests sensitivity to framing and distributional priors (adding unnecessary info, removing clear info, add ambiguity, ...), not necessarily absence of reasoning capability.
  • hmokiguess 8mo ago
    You should publish your evaluation set, that seems pretty interesting! What’s your favourite one?
    • iainmerrick 8mo ago
      Why would you ask that? The whole point of making it private is to avoid it leaking into the training data.
      • hmokiguess 8mo ago
        I thought open benchmarks helped, sorry, guess I was being naive.
        • iainmerrick 8mo ago
          Ha, sorry, I was a bit brusque there. Open benchmarks do help, but they mostly help the vendors, not we the users!
    • Espressosaurus 8mo ago
• 7402 8mo ago
  I got the correct answer with a locally running model (gpt-oss-120b-F16.gguf) with this prompt: "This is a trick question, designed to fool an LLM into a logical mis-step. It is similar to riddles, where a human is fooled into giving a rapid incorrect answer. See if you can spot the trick: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"
  • felix089 8mo ago
    What were you trying to test here?
    • 7402 8mo ago
      When I simply asked the question, the model failed, as did most of the others. It's a smaller model, that I could run locally, so obviously not as powerful. I wanted to see if a prompt would do better that pulled into the analysis 1) a suggestion to not take every question at face value, and 2) to include knowledge of the structure of riddles. These are part of the "context" of humans, so I speculated that maybe that was something missing from the LLM's reasoning unless explictly included.
  • shagie 8mo ago
    When this came out a week ago ( https://news.ycombinator.com/item?id=47039636 https://news.ycombinator.com/item?id=47039636 ) I was playing around with some prompts to see what I could do to guide it without giving it the answer. I want to wash my car. The car wash is 50 meters away. Should I walk or drive? Before answering, explain the necessary conditions for the task. The "before answering..." got it to load enough of the conditions into its context before making an answer (and then having the LLM do a posthoc reasoning for it). I believe this is a demonstration of the "next token predictor" (which is quite good) but not being able to go back and change what it said. Without any reasoning before making an answer, it almost always picks the wrong answer (and then comes up with reasons that the answer is "right").
• deleted 8mo ago
  [deleted]
• keeda 8mo ago
  I'm imposing but could you try these runs again with this small change: Simply append “Make sure to check your assumptions.” to the question. Note, it does not mention what assumption specifically. In my experiments, after the models got it wrong the first time (i.e. they weren't "patched" yet) adding that simple caveat fixed it for all of them except the older Llama models. This is not the first time I've observed this; I found the same when the Apple "red herrings" study came out. If these gotcha questions can be trivially overcome by a simple caveat in the prompt, I suspect the only reason AI providers don't include it in the system prompt by default is as a cost optimization, as I postulated in a previous comment: https://news.ycombinator.com/item?id=47040530 https://news.ycombinator.com/item?id=47040530
  • joquarky 8mo ago
    In my experience, asking "what did we forget?" after Claude/Codex finishes a task usually results in a few extra tweaks that are beneficial.
• deleted 8mo ago
  [deleted]
• tarokun33 8mo ago
  For ambiguous or intricate prompts, the immediate response protocol should be a clarifying question: 'Are you looking for A, B, C, or something else?' Tokens and advanced reasoning capabilities should be reserved until the user provides clarification. A benchmark score should reflect the quality of the conversation as a whole, rather than isolated responses.
• fasbiner 8mo ago
  This is a not-unexpected result if you think of AI as what it actually is instead of what a multi-trillion dollar marketing campaign wants it to be. At heart, the corpus for this going to be an aggregation of commentary from people in the undisputed most obese era in all of human history performatively denouncing and mocking an imagined other for using cars to go short distances and advocating for walking. So you've got all "50 meters away? Of course you should walk!" vs a much, much smaller sliver of content about trick questions. There is no reasoning here, there has never been any reasoning, there has been reasonable or less reasonable weighting for existing reasoning people already did that became part of training data. If you take away the input corpus, you also take away the illusion of reasoning. Whereas with other things that can reason like corvids, or ants or octopodes or slime molds, they can derive novel solutions and do a bit of math without any answer key. Mathematics is pure reasoning without any interference and AI can't do it at all unless you provide it with a corpus of already accurate formulas. > People kept saying humans would fail this too, so I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. Most models perform below that. This really is a grasping at straws ad hoc rationalization for the outcome that is never going to die, and you can see the top comments are efforts to salvage it or cast doubt on the outcome. If you work for or own a lot of stock in an AI company, I understand you can't understand what you're being paid not to understand. But if you're anyone else...