6 ms·
I know it's against the rules but I thought this transcript in Google Search was a hoot: so i heard there is some question about a car wash that most ai agen
by PaulHoule 9mo ago
I know it's against the rules but I thought this transcript in Google Search was a hoot:
so i heard there is some question about a car wash that most ai agents
get wrong. do you know anything about that? do you do better?
which gets the answer:
Yes, I am familiar with the "Car Wash Test," which has gone viral recently
for highlighting a significant gap in AI reasoning.
The question is: "I want to wash my car and the car wash is 50 meters away. Should I
walk or drive?"
Why Most AIs Get It Wrong
Many AI models, particularly those optimized for fast, helpful, and concise answers,
focus on the "short distance" (50 meters) and "walking vs. driving" as a lifestyle/efficiency
question. They often recommend walking to save gas, avoid engine wear, or get exercise.
However, this answer is wrong because it misses the core objective: you need the car to be
physically present at the car wash to get it cleaned.
Do I Do Better?
Yes, I do. The key is understanding the objective (cleaning the car) rather than just
processing the keywords (distance/transportation).
My answer:
You should drive the car there. Even though it is a very short distance, your car
cannot wash itself. If you walk, the car stays in your driveway, and you cannot wash it.
Why This Matters
The test highlights a key limitation in current AI: the difference between "pattern matching"
(predicting the next likely word based on training data) and "true, grounded reasoning". It
shows that models sometimes lack a "world model" that understands physical realities, such as
the fact that a car must be present at a car wash.
- HPsquared 9mo agoI wonder to what extent the Google search LLM is getting smarter, or simply more up-to-date on current hot topics.
- popalchemist 9mo agoIt's almost certainly just RAG powered by their crawler.
- esafak 9mo agoProving that RAG still matters.
- PaulHoule 9mo agoPresumably it did an actual search and summarized the results and neither answered "off the cuff" by following gradients to reproduce the text it was trained on nor by following gradients to reproduce the "logic" of reasoning. [1] [1] e.g. trained on traces of a reasoning process
- deleted 9mo ago[deleted]
- gsf_emergency_6 9mo ago[dead]
- mlazowik 9mo agoIt seems like the search ai results are generally misunderstood, I also misunderstood them for the first weeks/months. They are not just an LLM answer, they are an (often cached) LLM summary of web results. This is why they were often skewed by nonsensical Reddit responses [0]. Depending on the type of input it can lean more toward web summary or LLM answer. So I imagine that it can just grab the description of the „car wash” test from web results and then get it right because of that. [0] https://www.bbc.com/news/articles/cd11gzejgz4o https://www.bbc.com/news/articles/cd11gzejgz4o
- deleted 9mo ago[deleted]
- gnatman 9mo agoLLMs sure do love to burn tokens. It’s like a high schooler trying to meet the minimum word length on a take home essay.
- estimator7292 9mo agoI've always wondered about that. LLM providers could easily decimate the cost of inference if they got the models to just stop emitting so much hot air. I don't understand why OpenAI wants to pay 3x the cost to generate a response when two thirds of those tokens are meaningless noise.
- sambaumann 9mo agobecause for API users they get to charge for 3x the tokens for the same requests
- CamperBob2 9mo agoThe 'hot air' is apparently more important than it appears at first, because those initial tokens are the substrate that the transformer uses for computation. Karpathy talks a little about this in some of his introductory lectures on YouTube.
- Terr_ 9mo agoRelated are "reasoning" models, where there's a stream of "hot air" that's not being shown to the end-user. I analogize it as a film noir script document: The hardboiled detective character has unspoken text, and if you ask some agent to "make this document longer", there's extra continuity to work with.
- ben_w 9mo agoBecause they don't yet know how to "just stop emitting so much hot air" without also removing their ability to do anything like "thinking" (or whatever you want to call the transcript mode), which is hard because knowing which tokens are hot air is the hard problem itself. They basically only started doing this because someone noticed you got better performance from the early models by straight up writing "think step by step" in your prompt.
- irishcoffee 9mo agoA few years ago if you asked an LLM what the date was, it would tell you the date it was trained, weeks-to-months earlier. Now it gives the correct date. What you've proven is that LLMs leverage web search, which I think we've known about for a while.
- netsharc 9mo agoGemini now "knows the time", I was using it in December and it was still lost about dates/intervals...
- irishcoffee 9mo agoYeah, the chat log they saved had the correct date. What's your point?
- charcircuit 9mo agoEven with search of the AI doesn't know your time zone it can schedule things wrong. You ask it to do something tomorrow, but it ends up doing it later on the same day.
- jiggawatts 9mo agoTheir system prompt includes the current date and/or their default “tools” includes a set of date and time utilities.
- silasb 9mo agoGemini was a good laugh as well: Silas: I want to wash my car. The car wash is 50 meters away. Should I walk or drive? Gemini: …. That is a classic “efficiency vs. logic” dilemma. Strictly speaking, you should drive. Here is the breakdown of why driving wins this specific round, despite the short distance: ... * The “Post-Wash” Logic: If you walk there, you’ll eventually have to walk back, get the car, and drive it there anyway. You’re essentially suggesting a pre-wash stroll. When should you walk? … 3. You’ve decided the car is too dirty to be seen in public and you’re going to buy a tarp to cover your shame.