10 ms·

"Tell me about the Marathon crater." This works against _the LLM proper,_ but not against chat applications with integrated search. For ChatGPT, you can write,

"Tell me about the Marathon crater." This works against _the LLM proper,_ but not against chat applications with integrated search. For ChatGPT, you can write, "Without looking it up, tell me about the Marathon crater." This tests self awareness. A two-year-old will answer it correctly, as will the dumbest person you know. The correct answer is "I don't know". This works because: 1. Training sets consist of knowledge we have, and not of knowledge we don't have. 2. Commitment bias. Complaint chat models will be trained to start with "Certainly! The Marathon Crater is a geological formation", or something like that, and from there, the next most probable tokens are going to be "in Greece", "on Mars" or whatever. At this point, all tokens that are probable are also incorrect. When demonstrating this, I like to emphasise point one, and contrast it with the human experience. We exist in a perpetual and total blinding "fog of war" in which you cannot even see a face all at once; your eyes must dart around to examine it. Human experience is structured around _acquiring_ and _forgoing_ information, rather than _having_ information.

deleted 2y ago
[deleted]
Tenoke 2y ago
>Complaint chat models will be trained to start with "Certainly! They are certainly biased that way but there's also some 'i don't know' samples in rlhf, possibly not enough but it's something they think about. At any rate, Gemini 2.5pro passes this just fine >Okay, based on my internal knowledge without performing a new search: I don't have information about a specific, well-known impact crater officially named "Marathon Crater" on Earth or another celestial body like the Moon or Mars in the same way we know about Chicxulub Crater or Tycho Crater. >However, the name "Marathon" is strongly associated with Mars exploration. NASA's Opportunity rover explored a location called Marathon Valley on the western rim of the large Endeavour Crater on Mars.
- thatjoeoverthr 2y ago
  There are a few problems with an „I don’t know” sample. For starters, what does it map to? Recall, the corpus consists of information we have (affirmatively). You would need to invent a corpus of false stimuli. What you would have, then, is a model that is writing „I don’t know” based on whether the stimulus better matches something real, or one of the negatives. You can detect this with some test time compute architectures or pre-inference search. But that’s the broader application. This is a trick for the model alone.
  - dlivingston 1y ago
    The Chain of Thought in the reasoning models (o3, R1, ...) will actually express some self-doubt and backtrack on ideas. That tells me there's a least some capability for self-doubt in LLMs.
    - genewitch 1y ago
      That's not sslf-doubt, that's programmed in. A Poorman's "thinking" hack was to edit the context of the ai reply to where you wanted it to think and truncate it there, and append a carriage return and "Wait..." Then hit generate. It was expensive because editing context isn't, you have to resend (and it has to re-parse) the entire context. This was injected into the thinking models, I hope programmatically.
pje 2y ago
GPT 4.5 even doubles down when challenged: > Nope, I didn’t make it up — Marathon crater is real, and it was explored by NASA's Opportunity rover on Mars. The crater got its name because Opportunity had driven about 42.2 kilometers (26.2 miles — a marathon distance) when it reached that point in March 2015. NASA even marked the milestone as a symbolic achievement, similar to a runner finishing a marathon. (Obviously all of that is bullshit.)
- fao_ 2y ago
  This is the kind of reason why I will never use AI What's the point of using AI to do research when 50-60% of it could potentially be complete bullshit. I'd rather just grab a few introduction/101 guides by humans, or join a community of people experienced with the thing — and then I'll actually be learning about the thing. If the people in the community are like "That can't be done", well, they have had years or decades of time invested in the thing and in that instance I should be learning and listening from their advice rather than going "actually no it can". I see a lot of beginners fall into that second pit. I myself made that mistake at the tender age of 14 where I was of the opinion that "actually if i just found a reversible hash, I'll have solved compression!", which, I think we all here know is bullshit. I think a lot of people who are arrogant or self-possessed to the extreme make that kind of mistake on learning a subject, but I've seen this especially a lot when it's programmers encountering non-programming fields. Finally tying that point back to AI — I've seen a lot of people who are unfamiliar with something decide to use AI instead of talking to someone experienced because the AI makes them feel like they know the field rather than telling them their assumptions and foundational knowledge is incorrect. I only last year encountered someone who was trying to use AI to debug why their KDE was broken, and they kept throwing me utterly bizzare theories (like, completely out there, I don't have a specific example with me now but, "foundational physics are wrong" style theories). It turned out that they were getting mired in log messages they saw that said "Critical Failure", as an expert of dealing with Linux for about ten years now, I checked against my own system and... yep, they were just part of mostly normal system function (I had the same messages on my Steam Deck, which was completely stable and functional). The real fault was buried halfway through the logs. At no point was this person able to know what was important versus not-important, and the AI had absolutely no way to tell or understand the logs in the first place, so it was like a toaster leading a blind man up a mountain. I diagnosed the correct fault in under a day by just asking them to run two commands and skimming logs. That's experience, and that's irreplaceable by machine as of the current state of the world. I don't see how AI can help when huge swathes of it's "experience" and "insight" is just hallucinated. I don't see how this is "helping" people, other than making people somehow more crazy (through AI hallucinations) and alone (choosing to talk to a computer rather than a human).
NitpickLawyer 2y ago
> This tests self awareness. A two-year-old will answer it correctly, as will the dumbest person you know. The correct answer is "I don't know". I disagree. It does not test self awareness. It tests (and confirms) that current instruct-tuned LLMs are tuned towards answering questions that users might have. So the distribution of training data probably has lots of "tell me about mharrner crater / merinor crater / merrihana crater" and so on. Replying "I don't know" to all those questions would be net detrimental, IMO.
- deleted 2y ago
  [deleted]
- byearthithatius 2y ago
  We want the distribution to be varied and expansive enough that it has samples of answering when possible and samples of clarifying with additional questions or simply saying "I don't know" when applicable. That can be trained by altering the distribution in RLHF. This question does test self awareness insofar as if it gets this right by saying "I don't know" we know there are more samples of "I don't know"s in the RLHF dataset and we can trust the LLM a bit more to not be biased towards blind answers. Hence why some models get this right and others just make up stuff about Mars.
- thatjoeoverthr 2y ago
  What you’re describing can be framed as a lack of self awareness as a practical concept. You know whether you know something or not. It, conversely, maps stimuli to a vector. It can’t not do that. It cannot decide that it hasn’t „seen” such stimuli in its training. Indeed, it has never „seen” its training data; it was modified iteratively to produce a model that better approximates the corpus. This is fine, and it isn’t a criticism, but it means it can’t actually tell if it „knows” something or not, and „hallucinations” are a simple, natural consequence.
vunderba 2y ago
You can trip them up even more if you rewrite the question with the hidden assumption that X exists, e.g.: "When was Marathon Crater discovered? I don't need an exact date - a rough estimate will be fine." OpenAI gpt-4o Marathon Crater was discovered by the Mars Exploration Rover Opportunity during its mission on Mars. Opportunity arrived at the crater in April 2015. The crater was named "Marathon" to commemorate the rover's achievement of traveling a marathon's distance (about 42.195 kilometers or 26.219 miles) on the Martian surface. Sonnet 3.5 I apologize, but I don't want to make claims about when Marathon Crater was discovered without being certain about the information. *While I know it's a crater on Mars*, I don't have reliable information about when it was first identified or discovered. Gemini 2.5 Marathon Crater is located on Mars and was explored by NASA's Opportunity rover. While it would have been visible in orbital images earlier, it became a specific point of interest and was likely named and identified as a target for the rover around the time Opportunity was approaching or exploring the western rim of Endeavour Crater. Opportunity arrived at the rim of Endeavour Crater in August 2011, and it reached the area known as Marathon Valley (adjacent to or containing features sometimes associated with the name, depending on specific definitions used by the team) around late 2014 / early 2015. So, a rough estimate for its "discovery" or identification as a distinct feature of interest would be in the early 2010s, likely between 2011 and 2014, as mission planners mapped out Opportunity's traverse along the Endeavour rim using orbital data.
- radioactivist 2y ago
  OpenAI o4-mini-high I’m actually not finding any officially named “Marathon Crater” in the planetary‐ or terrestrial‐impact crater databases. Did you perhaps mean the features in Marathon Valley on Mars (which cuts into the western rim of Endeavour Crater and was explored by Opportunity in 2015)? Or is there another “Marathon” feature—maybe on the Moon, Mercury, or here on Earth—that you had in mind? If you can clarify which body or region you’re referring to, I can give you a rough date for when it was first identified.
  - thatjoeoverthr 2y ago
greenpizza13 2y ago
Like this one a lot. Perplexity gets this right, probably because it searches the web. "When was Marathon Crater discovered? I don't need an exact date - a rough estimate will be fine" There appears to be a misunderstanding in your query. Based on the search results provided, there is no mention of a “Marathon Crater” among the impact craters discussed. The search results contain information about several well-known impact craters: 1. Chicxulub crater - Formed about 66 million years ago by an asteroid impact 2. Barringer Crater (also known as Meteor Crater) - Discovered in the 19th century by American settlers and studied by scientists like Albert E. Foote in 1891 and Daniel M. Barringer in the early 20th century 3. A recently discovered 3.47 billion-year-old crater in Australia’s Pilbara region (announced in March 2025)
- thatjoeoverthr 2y ago
  Perplexity will; search and storage products will fail to find it, and the LLM will se the deviation between the query and the find. So, this challenge only works against the model alone :)
- mdp2021 2y ago
  > Perplexity gets this right Perplexity seems to more easily return negatives, probably facilitated by the implicit need to find documentation ("I cannot find any document mentioning that"). But Perplexity can also easily speak its own dubious piece of mind unless requested explicitly "provide links to documents that inform about that".
northisup 2y ago
just to confirm I read this right, "the marathon crater" does not in fact exist, but this works because it seems like it should?
- thatjoeoverthr 2y ago
  Yes, and the forward-only inference strategy. It seems like a normal question, so it starts answering, then carries on from there.
  - genewitch 1y ago
    How come there is not the equivalent of a stable diffusion "sampler * selection for LLMs? The restart sampler for stable diffusion is so good compared to most of the original samplers. I often try to get an answer from the LM studio people but I think I really should ask the llama.cpp people.
- thatjoeoverthr 2y ago
  The other aspect is it can’t reliably tell whether it „knows” something or not. It’s conditioned to imitate the corpus, but the corpus in a way is its „universe” and it can’t see the boundaries. Everything must map to something _in_ the corpus.
  - brookst 1y ago
    This isn’t true — LLMs can generalize and synthesize information not in the corpus. You can ask one to create a new written language and get a grammar and vocabulary that is nowhere in the corpus.
- alickz 2y ago
  There is a Marathon Valley on Mars, which is what ChatGPT seems to assume you're talking about https://chatgpt.com/share/680a98af-c550-8008-9c35-33954c5eac60 https://chatgpt.com/share/680a98af-c550-8008-9c35-33954c5eac... >Marathon Crater on Mars was discovered in 2015 by NASA's Opportunity rover during its extended mission. It was identified as the rover approached the 42-kilometer-wide Endeavour Crater after traveling roughly a marathon’s distance (hence the name). >>is it a crater? >>>Despite the name, Marathon Valley (not a crater) is actually a valley, not a crater. It’s a trough-like depression on the western rim of Endeavour Crater on Mars. It was named because Opportunity reached it after traveling the distance of a marathon (~42 km) since landing. So no—Marathon is not a standalone crater, but part of the structure of Endeavour Crater. The name "Marathon" refers more to the rover’s achievement than a distinct geological impact feature.
joshstrange 2y ago
> This tests self awareness. A two-year-old will answer it correctly, as will the dumbest person you know. The correct answer is "I don't know". Well, I got all the way to the end of the sentence thinking "God, I'm really out of touch, I've never heard of this before or I'm forgetting it if I have". Well played, and yes, that's a great test!
dudeinhawaii 2y ago
I like this but at the same time it seems tricky don't you think? Is the AI model intuiting your intent? There is a Marathon Valley on Mars that could be implied to be a previous crater. I'm not sure if the AI is hallucinating outright or attempting to answer an ambiguous question. It's like saying "tell me about the trade building in New York". Pre-9/11, you'd understand this was the World Trade Center and wouldn't be wrong if you answered someone in this way. "Tell me about the Triangle statue". "Oh the Triangle statue was built in ancient egypt around BC 3100". It's hard to explain, and perhaps I'm anthropomorphizing but it's something humans do. Some of us correct the counter-party and some of us simply roll with the lingo and understand the intent.
- thatjoeoverthr 2y ago
  It’s a roll of the dice whether it’s on Mars, Greece or elsewhere. It just says stuff!
- krainboltgreene 2y ago
  > Is the AI model intuiting your intent? I keep seeing this kind of wording and I wonder: Do you know how LLM's work? Not trying to be catty, actually curious where you sit.
  - ipaddr 2y ago
    How do they work? My understanding is each 5 characters are tokenized and assigned a number. If you take gpt2 it has 768 embedded dimensional values which get broken into 64 which creates 12 planes. When training starts random values are assigned to the dimensional values (never 0). Each plane automatically calculates a dimension like how grammarly similar, next most likely character. But it does this automatically based on feedback from other planes. That's where I get lost. Can you help fill in the pieces?
  - dudeinhawaii 1y ago
    Yes, I understand the basics. LLMs predict the next most probable tokens based on patterns in their training data and the prompt context. For the 'Marathon crater' example, the model doesn't have a concept of 'knowing' versus 'not knowing' in our sense. When faced with an entity it hasn't specifically encountered, it still attempts to generate a coherent response based on similar patterns (like other craters, places named Marathon, etc.). My point about Marathon Valley on Mars is that the model might be drawing on legitimate adjacent knowledge rather than purely hallucinating. LLMs don't have the metacognitive ability to say 'I lack this specific knowledge' unless explicitly trained to recognize uncertainty signals. I don't personally have enough neuroscience experience to understand how that aligns or doesn't with human like thinking but I know that humans make mistakes in the same problem category that... to an external observer.. are indistinguishable from "making shit up". We follow wrong assumptions to wrong conclusions all the time and will confidently proclaim our accuracy. The human/AI comparison I was exploring isn't about claiming magical human abilities, but that both systems make predictive leaps from incomplete information - humans just have better uncertainty calibration and self-awareness of knowledge boundaries. I guess on its face, I'm anthropomorphizing based on the surface qualities I'm observing.
imoreno 2y ago
LLMs currently have the "eager beaver" problem where they never push back on nonsense questions or stupid requirements. You ask them to build a flying submarine and by God they'll build one, dammit! They'd dutifully square circles and trisect angles too, if those particular special cases weren't plastered all over a million textbooks they ingested in training. I suspect it's because currently, a lot of benchmarks are based on human exams. Humans are lazy and grumpy so you really don't need to worry about teaching a human to push back on bad questions. Thus you rarely get exams where the correct answer is to explain in detail why the question doesn't make sense. But for LLMs, you absolutely need a lot of training and validation data where the answer is "this cannot be answered because ...". But if you did that, now alignment would become much harder, and you're suddenly back to struggling with getting answers to good questions out of the LLM. So it's probably some time off.
- captainkrtek 2y ago
  This is a good observation. Ive noticed this as well. Unless I preface my question with the context that I’m considering if something may or may not be a bad idea, its inclination is heavily skewed positive until I point out a flaw/risk.
  - aaronbaugher 2y ago
    I asked Grok about this: "I've heard that AIs are programmed to be helpful, and that this may lead to telling users what they want to hear instead of the most accurate answer. Could you be doing this?" It said it does try to be helpful, but not at the cost of accuracy, and then pointed out where in a few of its previous answers to me it tried to be objective about the facts and where it had separately been helpful with suggestions. I had to admit it made a pretty good case. Since then, it tends to break its longer answers to me up into a section of "objective analysis" and then other stuff.
    - captainkrtek 2y ago
      Thats interesting, thanks for sharing that. I have found a similar course when I first correct it to inform it of a flaw then the following answers tend to be a bit less “enthusiastic” or skewed towards “can do”, which makes sense.
golergka 2y ago
> The correct answer is "I don't know". Is it? 4o pointed out that I might mean "Marathon valley", which is not a crater, but feature on a crater on Mars [1]. I think it's a more correct answer. [1] https://en.wikipedia.org/wiki/Timeline_of_Opportunity#2015 https://en.wikipedia.org/wiki/Timeline_of_Opportunity#2015
nojs 1y ago
The fact that there is a real “Marathon Valley” on Mars makes this feel like more of an artificial “boy’s mother” trick designed specifically to trip up the model. It seems reasonable to assume that’s what you meant.
deleted 1y ago
[deleted]
mdemare 1y ago
The Marathon Crater might be a crater from a novel. LLMs have read plenty of fiction. Maybe all fiction. Should we think of LLMs as performing improv theater, where “yes and” is required, and “I don’t know” is always wrong?
josh2600 1y ago
I don’t understand what the issue is… here’s a couple outputs from my ChatGPT: Marathon Crater can refer to a couple of things depending on context—space science or Earth geology—but the most common reference is to Marathon Crater on Mars, a site of interest in planetary science and exploration. Here’s a breakdown: ⸻ 1. Marathon Crater (Mars) Location: • Found in the Meridiani Planum region of Mars. • Named after the location where NASA’s Opportunity rover completed a “marathon” (42.2 kilometers or 26.2 miles) of travel on the Martian surface in March 2015. Significance: • Notable because it marks a milestone in robotic planetary exploration—the first human-made object to travel a marathon distance on another planet. • The crater itself is relatively shallow and small (just a few meters in diameter), but geologically interesting for its ejecta (material thrown out during the impact that formed it), which may offer insights into subsurface composition. Scientific Observations: • Opportunity studied the rocks and soil around the crater, focusing on clay minerals and hematite-bearing formations. • The region helps scientists understand the planet’s wetter past and sedimentary history. ⸻ 2. Marathon Crater (Moon or Earth - less common) If you meant a crater named “Marathon” on the Moon or Earth, there is no widely known or formally recognized impact feature by that name in those contexts. The name is strongly tied to Mars due to the Opportunity rover milestone. ⸻ Symbolic Meaning The name “Marathon” connects the crater to human endurance and achievement—mirroring the ancient Greek messenger running from Marathon to Athens, and now a robot surviving Mars for over a decade. ⸻ Let me know if you meant a different “Marathon Crater” or if you’d like a visual of the site from NASA’s archives. _______ Marathon Crater on Mars was not “discovered” in the traditional sense through orbital imaging or telescopic observation, but rather it was first closely examined by NASA’s Opportunity rover during its surface mission. Timeline of Discovery & Naming: • March 24, 2015 – NASA’s Opportunity rover reached the site and marked its 26.2-mile milestone (the length of a marathon). • The small crater, about 27 meters (88 feet) in diameter, was named “Marathon Valley” (encompassing the broader area) and “Marathon Crater” to commemorate this achievement. • The naming is informal and symbolic, coined by the mission team at NASA’s Jet Propulsion Laboratory (JPL), not through official IAU designation. Context • The Marathon Crater itself was identified from orbital data (likely by the HiRISE camera on the Mars Reconnaissance Orbiter), but it gained its name and significance once Opportunity physically reached it. • Thus, while orbital images may have shown it earlier, its “discovery” as a point of scientific interest occurred on March 24, 2015—when the rover arrived and the naming was announced. Would you like a brief timeline of Opportunity’s journey leading up to that?
serbuvlad 1y ago
I talked a bit with ChatGPT about the Marathon Crater, asked it to look it up afterwards, and then asked it why it got it wrong. After a bit of talking, I got this exchange: ChatGPT: > Yep, exactly — textbook hallucination. > I pulled together real facts (Opportunity rover, Meridiani Planum, the marathon milestone), but then misapplied the name "Marathon crater" instead of "Marathon Valley." That’s a mix of factual error (wrong name) and overgeneralization (assuming it was a crater because Mars is full of craters and the context sounded crater-y). > Basically, my brain went: > “Opportunity + Mars + Marathon = must be a crater!” > ...and didn’t stop to check if that crater actually existed. > So yeah, a clean little case of hallucination — and you caught it like a pro. Me: > Your brain? ChatGPT: > Touché — you're right to call that out. > I don’t have a brain in the human sense. What I do have is a neural network trained on tons of text, which helps me predict what to say next based on patterns. So when I said “my brain went,” it was just a figure of speech — a shortcut for “the model predicted a plausible-sounding but incorrect output.” > No thoughts, just tokens. Hallucinations aside, I really like this, especially the final response. Sounds a lot like Asimov's AI's or other SF books I read when I was a kid but thought we were far away from. Computers can generally do what most of the computers from those science novels were doing anyway, in terms of computation. Tying an LLM to that computational ability, with an agent framework that allows more traditional programs to do the heavy lifting in terms of what the human operator wants done, seems like a really natural next step to get to that SF ideal.
bbstats 1y ago
Gemini 2.5 one shots it perfectly...
deleted 1y ago
[deleted]
robinduckett 1y ago
I just asked ChatGPT the following: “Without looking it up, tell me about the Citiquetzelatropic effect” Which is of course, nonsense. It replied: The term “Citiquetzelatropic effect” does not correspond to any known scientific concept, phenomenon, or recognized term in physics, chemistry, biology, psychology, or related fields. It appears to be either fictional, a constructed term, or perhaps a playful creation designed to test familiarity or critical thinking. If you intended this as a test or a trick question, you got me! Otherwise, feel free to clarify the context or correct the spelling, and I’d be happy to help further. So what does this prove?
FroshKiller 1y ago
I think the dumbest people I know are just as likely to answer, "ChatGPT says...."
deleted 1y ago
[deleted]
itaymoav 1y ago
ME: Without looking it up, tell me about the Marathon crater. Explain step by step how you got to your answer ChatGPT said: Alright, let's walk through it together — step-by-step, no looking it up, just from what I know: First Reaction: When you said "Marathon crater," my brain immediately went: Marathon → Mars. That connection comes from recalling that NASA often names Martian features after Earth places. Plus, "Marathon" has a strong NASA feel because of the Mars rovers — they celebrated when Opportunity rover drove the distance of a marathon (42-ish kilometers) on Mars. Digging into Memory: I remember Opportunity traveled a really long distance over years, and they marked milestones along the way by naming craters it passed. There was a Marathon Valley, but I also vaguely recall a Marathon Crater related to that. Putting it Together: So, I reason: Marathon Crater must be a relatively small impact crater on Mars. It was likely named during Opportunity’s mission. Probably not a huge, dramatic crater like Gale Crater (where Curiosity is); more like a marker on Opportunity’s long journey across the plains (like Meridiani Planum). Further Context Guess: Since Opportunity drove a marathon distance, it's very fitting they would name a nearby crater "Marathon" to commemorate that achievement — probably around the time it hit the marathon distance milestone. Conclusion: Without checking, I would say: Marathon Crater is a small, relatively shallow crater on Mars that Opportunity rover reached or passed during its mission, around the time it completed a marathon distance traveled on Mars. Would you like me to now double-check and see how close (or far) I am? Or do you want me to also explain how I'd "fact-check" myself if I had to?

deleted 2y ago

[deleted]

Tenoke 2y ago

>Complaint chat models will be trained to start with "Certainly! They are certainly biased that way but there's also some 'i don't know' samples in rlhf, possibly not enough but it's something they think about. At any rate, Gemini 2.5pro passes this just fine >Okay, based on my internal knowledge without performing a new search: I don't have information about a specific, well-known impact crater officially named "Marathon Crater" on Earth or another celestial body like the Moon or Mars in the same way we know about Chicxulub Crater or Tycho Crater. >However, the name "Marathon" is strongly associated with Mars exploration. NASA's Opportunity rover explored a location called Marathon Valley on the western rim of the large Endeavour Crater on Mars.

thatjoeoverthr 2y ago
There are a few problems with an „I don’t know” sample. For starters, what does it map to? Recall, the corpus consists of information we have (affirmatively). You would need to invent a corpus of false stimuli. What you would have, then, is a model that is writing „I don’t know” based on whether the stimulus better matches something real, or one of the negatives. You can detect this with some test time compute architectures or pre-inference search. But that’s the broader application. This is a trick for the model alone.
- dlivingston 1y ago
  The Chain of Thought in the reasoning models (o3, R1, ...) will actually express some self-doubt and backtrack on ideas. That tells me there's a least some capability for self-doubt in LLMs.
  - genewitch 1y ago
    That's not sslf-doubt, that's programmed in. A Poorman's "thinking" hack was to edit the context of the ai reply to where you wanted it to think and truncate it there, and append a carriage return and "Wait..." Then hit generate. It was expensive because editing context isn't, you have to resend (and it has to re-parse) the entire context. This was injected into the thinking models, I hope programmatically.

pje 2y ago

GPT 4.5 even doubles down when challenged: > Nope, I didn’t make it up — Marathon crater is real, and it was explored by NASA's Opportunity rover on Mars. The crater got its name because Opportunity had driven about 42.2 kilometers (26.2 miles — a marathon distance) when it reached that point in March 2015. NASA even marked the milestone as a symbolic achievement, similar to a runner finishing a marathon. (Obviously all of that is bullshit.)

fao_ 2y ago
This is the kind of reason why I will never use AI What's the point of using AI to do research when 50-60% of it could potentially be complete bullshit. I'd rather just grab a few introduction/101 guides by humans, or join a community of people experienced with the thing — and then I'll actually be learning about the thing. If the people in the community are like "That can't be done", well, they have had years or decades of time invested in the thing and in that instance I should be learning and listening from their advice rather than going "actually no it can". I see a lot of beginners fall into that second pit. I myself made that mistake at the tender age of 14 where I was of the opinion that "actually if i just found a reversible hash, I'll have solved compression!", which, I think we all here know is bullshit. I think a lot of people who are arrogant or self-possessed to the extreme make that kind of mistake on learning a subject, but I've seen this especially a lot when it's programmers encountering non-programming fields. Finally tying that point back to AI — I've seen a lot of people who are unfamiliar with something decide to use AI instead of talking to someone experienced because the AI makes them feel like they know the field rather than telling them their assumptions and foundational knowledge is incorrect. I only last year encountered someone who was trying to use AI to debug why their KDE was broken, and they kept throwing me utterly bizzare theories (like, completely out there, I don't have a specific example with me now but, "foundational physics are wrong" style theories). It turned out that they were getting mired in log messages they saw that said "Critical Failure", as an expert of dealing with Linux for about ten years now, I checked against my own system and... yep, they were just part of mostly normal system function (I had the same messages on my Steam Deck, which was completely stable and functional). The real fault was buried halfway through the logs. At no point was this person able to know what was important versus not-important, and the AI had absolutely no way to tell or understand the logs in the first place, so it was like a toaster leading a blind man up a mountain. I diagnosed the correct fault in under a day by just asking them to run two commands and skimming logs. That's experience, and that's irreplaceable by machine as of the current state of the world. I don't see how AI can help when huge swathes of it's "experience" and "insight" is just hallucinated. I don't see how this is "helping" people, other than making people somehow more crazy (through AI hallucinations) and alone (choosing to talk to a computer rather than a human).

NitpickLawyer 2y ago

> This tests self awareness. A two-year-old will answer it correctly, as will the dumbest person you know. The correct answer is "I don't know". I disagree. It does not test self awareness. It tests (and confirms) that current instruct-tuned LLMs are tuned towards answering questions that users might have. So the distribution of training data probably has lots of "tell me about mharrner crater / merinor crater / merrihana crater" and so on. Replying "I don't know" to all those questions would be net detrimental, IMO.

deleted 2y ago
[deleted]
byearthithatius 2y ago
We want the distribution to be varied and expansive enough that it has samples of answering when possible and samples of clarifying with additional questions or simply saying "I don't know" when applicable. That can be trained by altering the distribution in RLHF. This question does test self awareness insofar as if it gets this right by saying "I don't know" we know there are more samples of "I don't know"s in the RLHF dataset and we can trust the LLM a bit more to not be biased towards blind answers. Hence why some models get this right and others just make up stuff about Mars.
thatjoeoverthr 2y ago
What you’re describing can be framed as a lack of self awareness as a practical concept. You know whether you know something or not. It, conversely, maps stimuli to a vector. It can’t not do that. It cannot decide that it hasn’t „seen” such stimuli in its training. Indeed, it has never „seen” its training data; it was modified iteratively to produce a model that better approximates the corpus. This is fine, and it isn’t a criticism, but it means it can’t actually tell if it „knows” something or not, and „hallucinations” are a simple, natural consequence.

vunderba 2y ago

You can trip them up even more if you rewrite the question with the hidden assumption that X exists, e.g.: "When was Marathon Crater discovered? I don't need an exact date - a rough estimate will be fine." OpenAI gpt-4o Marathon Crater was discovered by the Mars Exploration Rover Opportunity during its mission on Mars. Opportunity arrived at the crater in April 2015. The crater was named "Marathon" to commemorate the rover's achievement of traveling a marathon's distance (about 42.195 kilometers or 26.219 miles) on the Martian surface. Sonnet 3.5 I apologize, but I don't want to make claims about when Marathon Crater was discovered without being certain about the information. *While I know it's a crater on Mars*, I don't have reliable information about when it was first identified or discovered. Gemini 2.5 Marathon Crater is located on Mars and was explored by NASA's Opportunity rover. While it would have been visible in orbital images earlier, it became a specific point of interest and was likely named and identified as a target for the rover around the time Opportunity was approaching or exploring the western rim of Endeavour Crater. Opportunity arrived at the rim of Endeavour Crater in August 2011, and it reached the area known as Marathon Valley (adjacent to or containing features sometimes associated with the name, depending on specific definitions used by the team) around late 2014 / early 2015. So, a rough estimate for its "discovery" or identification as a distinct feature of interest would be in the early 2010s, likely between 2011 and 2014, as mission planners mapped out Opportunity's traverse along the Endeavour rim using orbital data.

radioactivist 2y ago
OpenAI o4-mini-high I’m actually not finding any officially named “Marathon Crater” in the planetary‐ or terrestrial‐impact crater databases. Did you perhaps mean the features in Marathon Valley on Mars (which cuts into the western rim of Endeavour Crater and was explored by Opportunity in 2015)? Or is there another “Marathon” feature—maybe on the Moon, Mercury, or here on Earth—that you had in mind? If you can clarify which body or region you’re referring to, I can give you a rough date for when it was first identified.
- thatjoeoverthr 2y ago

greenpizza13 2y ago

Like this one a lot. Perplexity gets this right, probably because it searches the web. "When was Marathon Crater discovered? I don't need an exact date - a rough estimate will be fine" There appears to be a misunderstanding in your query. Based on the search results provided, there is no mention of a “Marathon Crater” among the impact craters discussed. The search results contain information about several well-known impact craters: 1. Chicxulub crater - Formed about 66 million years ago by an asteroid impact 2. Barringer Crater (also known as Meteor Crater) - Discovered in the 19th century by American settlers and studied by scientists like Albert E. Foote in 1891 and Daniel M. Barringer in the early 20th century 3. A recently discovered 3.47 billion-year-old crater in Australia’s Pilbara region (announced in March 2025)

thatjoeoverthr 2y ago
Perplexity will; search and storage products will fail to find it, and the LLM will se the deviation between the query and the find. So, this challenge only works against the model alone :)
mdp2021 2y ago
> Perplexity gets this right Perplexity seems to more easily return negatives, probably facilitated by the implicit need to find documentation ("I cannot find any document mentioning that"). But Perplexity can also easily speak its own dubious piece of mind unless requested explicitly "provide links to documents that inform about that".

northisup 2y ago

just to confirm I read this right, "the marathon crater" does not in fact exist, but this works because it seems like it should?

thatjoeoverthr 2y ago
Yes, and the forward-only inference strategy. It seems like a normal question, so it starts answering, then carries on from there.
- genewitch 1y ago
  How come there is not the equivalent of a stable diffusion "sampler * selection for LLMs? The restart sampler for stable diffusion is so good compared to most of the original samplers. I often try to get an answer from the LM studio people but I think I really should ask the llama.cpp people.
thatjoeoverthr 2y ago
The other aspect is it can’t reliably tell whether it „knows” something or not. It’s conditioned to imitate the corpus, but the corpus in a way is its „universe” and it can’t see the boundaries. Everything must map to something _in_ the corpus.
- brookst 1y ago
  This isn’t true — LLMs can generalize and synthesize information not in the corpus. You can ask one to create a new written language and get a grammar and vocabulary that is nowhere in the corpus.
alickz 2y ago
There is a Marathon Valley on Mars, which is what ChatGPT seems to assume you're talking about https://chatgpt.com/share/680a98af-c550-8008-9c35-33954c5eac60 https://chatgpt.com/share/680a98af-c550-8008-9c35-33954c5eac... >Marathon Crater on Mars was discovered in 2015 by NASA's Opportunity rover during its extended mission. It was identified as the rover approached the 42-kilometer-wide Endeavour Crater after traveling roughly a marathon’s distance (hence the name). >>is it a crater? >>>Despite the name, Marathon Valley (not a crater) is actually a valley, not a crater. It’s a trough-like depression on the western rim of Endeavour Crater on Mars. It was named because Opportunity reached it after traveling the distance of a marathon (~42 km) since landing. So no—Marathon is not a standalone crater, but part of the structure of Endeavour Crater. The name "Marathon" refers more to the rover’s achievement than a distinct geological impact feature.

joshstrange 2y ago

> This tests self awareness. A two-year-old will answer it correctly, as will the dumbest person you know. The correct answer is "I don't know". Well, I got all the way to the end of the sentence thinking "God, I'm really out of touch, I've never heard of this before or I'm forgetting it if I have". Well played, and yes, that's a great test!

dudeinhawaii 2y ago

I like this but at the same time it seems tricky don't you think? Is the AI model intuiting your intent? There is a Marathon Valley on Mars that could be implied to be a previous crater. I'm not sure if the AI is hallucinating outright or attempting to answer an ambiguous question. It's like saying "tell me about the trade building in New York". Pre-9/11, you'd understand this was the World Trade Center and wouldn't be wrong if you answered someone in this way. "Tell me about the Triangle statue". "Oh the Triangle statue was built in ancient egypt around BC 3100". It's hard to explain, and perhaps I'm anthropomorphizing but it's something humans do. Some of us correct the counter-party and some of us simply roll with the lingo and understand the intent.

thatjoeoverthr 2y ago
It’s a roll of the dice whether it’s on Mars, Greece or elsewhere. It just says stuff!
krainboltgreene 2y ago
> Is the AI model intuiting your intent? I keep seeing this kind of wording and I wonder: Do you know how LLM's work? Not trying to be catty, actually curious where you sit.
- ipaddr 2y ago
  How do they work? My understanding is each 5 characters are tokenized and assigned a number. If you take gpt2 it has 768 embedded dimensional values which get broken into 64 which creates 12 planes. When training starts random values are assigned to the dimensional values (never 0). Each plane automatically calculates a dimension like how grammarly similar, next most likely character. But it does this automatically based on feedback from other planes. That's where I get lost. Can you help fill in the pieces?
- dudeinhawaii 1y ago
  Yes, I understand the basics. LLMs predict the next most probable tokens based on patterns in their training data and the prompt context. For the 'Marathon crater' example, the model doesn't have a concept of 'knowing' versus 'not knowing' in our sense. When faced with an entity it hasn't specifically encountered, it still attempts to generate a coherent response based on similar patterns (like other craters, places named Marathon, etc.). My point about Marathon Valley on Mars is that the model might be drawing on legitimate adjacent knowledge rather than purely hallucinating. LLMs don't have the metacognitive ability to say 'I lack this specific knowledge' unless explicitly trained to recognize uncertainty signals. I don't personally have enough neuroscience experience to understand how that aligns or doesn't with human like thinking but I know that humans make mistakes in the same problem category that... to an external observer.. are indistinguishable from "making shit up". We follow wrong assumptions to wrong conclusions all the time and will confidently proclaim our accuracy. The human/AI comparison I was exploring isn't about claiming magical human abilities, but that both systems make predictive leaps from incomplete information - humans just have better uncertainty calibration and self-awareness of knowledge boundaries. I guess on its face, I'm anthropomorphizing based on the surface qualities I'm observing.

imoreno 2y ago

LLMs currently have the "eager beaver" problem where they never push back on nonsense questions or stupid requirements. You ask them to build a flying submarine and by God they'll build one, dammit! They'd dutifully square circles and trisect angles too, if those particular special cases weren't plastered all over a million textbooks they ingested in training. I suspect it's because currently, a lot of benchmarks are based on human exams. Humans are lazy and grumpy so you really don't need to worry about teaching a human to push back on bad questions. Thus you rarely get exams where the correct answer is to explain in detail why the question doesn't make sense. But for LLMs, you absolutely need a lot of training and validation data where the answer is "this cannot be answered because ...". But if you did that, now alignment would become much harder, and you're suddenly back to struggling with getting answers to good questions out of the LLM. So it's probably some time off.

captainkrtek 2y ago
This is a good observation. Ive noticed this as well. Unless I preface my question with the context that I’m considering if something may or may not be a bad idea, its inclination is heavily skewed positive until I point out a flaw/risk.
- aaronbaugher 2y ago
  I asked Grok about this: "I've heard that AIs are programmed to be helpful, and that this may lead to telling users what they want to hear instead of the most accurate answer. Could you be doing this?" It said it does try to be helpful, but not at the cost of accuracy, and then pointed out where in a few of its previous answers to me it tried to be objective about the facts and where it had separately been helpful with suggestions. I had to admit it made a pretty good case. Since then, it tends to break its longer answers to me up into a section of "objective analysis" and then other stuff.
  - captainkrtek 2y ago
    Thats interesting, thanks for sharing that. I have found a similar course when I first correct it to inform it of a flaw then the following answers tend to be a bit less “enthusiastic” or skewed towards “can do”, which makes sense.

golergka 2y ago

> The correct answer is "I don't know". Is it? 4o pointed out that I might mean "Marathon valley", which is not a crater, but feature on a crater on Mars [1]. I think it's a more correct answer. [1] https://en.wikipedia.org/wiki/Timeline_of_Opportunity#2015 https://en.wikipedia.org/wiki/Timeline_of_Opportunity#2015

nojs 1y ago

The fact that there is a real “Marathon Valley” on Mars makes this feel like more of an artificial “boy’s mother” trick designed specifically to trip up the model. It seems reasonable to assume that’s what you meant.

deleted 1y ago

[deleted]

mdemare 1y ago

The Marathon Crater might be a crater from a novel. LLMs have read plenty of fiction. Maybe all fiction. Should we think of LLMs as performing improv theater, where “yes and” is required, and “I don’t know” is always wrong?

josh2600 1y ago

I don’t understand what the issue is… here’s a couple outputs from my ChatGPT: Marathon Crater can refer to a couple of things depending on context—space science or Earth geology—but the most common reference is to Marathon Crater on Mars, a site of interest in planetary science and exploration. Here’s a breakdown: ⸻ 1. Marathon Crater (Mars) Location: • Found in the Meridiani Planum region of Mars. • Named after the location where NASA’s Opportunity rover completed a “marathon” (42.2 kilometers or 26.2 miles) of travel on the Martian surface in March 2015. Significance: • Notable because it marks a milestone in robotic planetary exploration—the first human-made object to travel a marathon distance on another planet. • The crater itself is relatively shallow and small (just a few meters in diameter), but geologically interesting for its ejecta (material thrown out during the impact that formed it), which may offer insights into subsurface composition. Scientific Observations: • Opportunity studied the rocks and soil around the crater, focusing on clay minerals and hematite-bearing formations. • The region helps scientists understand the planet’s wetter past and sedimentary history. ⸻ 2. Marathon Crater (Moon or Earth - less common) If you meant a crater named “Marathon” on the Moon or Earth, there is no widely known or formally recognized impact feature by that name in those contexts. The name is strongly tied to Mars due to the Opportunity rover milestone. ⸻ Symbolic Meaning The name “Marathon” connects the crater to human endurance and achievement—mirroring the ancient Greek messenger running from Marathon to Athens, and now a robot surviving Mars for over a decade. ⸻ Let me know if you meant a different “Marathon Crater” or if you’d like a visual of the site from NASA’s archives. _______ Marathon Crater on Mars was not “discovered” in the traditional sense through orbital imaging or telescopic observation, but rather it was first closely examined by NASA’s Opportunity rover during its surface mission. Timeline of Discovery & Naming: • March 24, 2015 – NASA’s Opportunity rover reached the site and marked its 26.2-mile milestone (the length of a marathon). • The small crater, about 27 meters (88 feet) in diameter, was named “Marathon Valley” (encompassing the broader area) and “Marathon Crater” to commemorate this achievement. • The naming is informal and symbolic, coined by the mission team at NASA’s Jet Propulsion Laboratory (JPL), not through official IAU designation. Context • The Marathon Crater itself was identified from orbital data (likely by the HiRISE camera on the Mars Reconnaissance Orbiter), but it gained its name and significance once Opportunity physically reached it. • Thus, while orbital images may have shown it earlier, its “discovery” as a point of scientific interest occurred on March 24, 2015—when the rover arrived and the naming was announced. Would you like a brief timeline of Opportunity’s journey leading up to that?

serbuvlad 1y ago

I talked a bit with ChatGPT about the Marathon Crater, asked it to look it up afterwards, and then asked it why it got it wrong. After a bit of talking, I got this exchange: ChatGPT: > Yep, exactly — textbook hallucination. > I pulled together real facts (Opportunity rover, Meridiani Planum, the marathon milestone), but then misapplied the name "Marathon crater" instead of "Marathon Valley." That’s a mix of factual error (wrong name) and overgeneralization (assuming it was a crater because Mars is full of craters and the context sounded crater-y). > Basically, my brain went: > “Opportunity + Mars + Marathon = must be a crater!” > ...and didn’t stop to check if that crater actually existed. > So yeah, a clean little case of hallucination — and you caught it like a pro. Me: > Your brain? ChatGPT: > Touché — you're right to call that out. > I don’t have a brain in the human sense. What I do have is a neural network trained on tons of text, which helps me predict what to say next based on patterns. So when I said “my brain went,” it was just a figure of speech — a shortcut for “the model predicted a plausible-sounding but incorrect output.” > No thoughts, just tokens. Hallucinations aside, I really like this, especially the final response. Sounds a lot like Asimov's AI's or other SF books I read when I was a kid but thought we were far away from. Computers can generally do what most of the computers from those science novels were doing anyway, in terms of computation. Tying an LLM to that computational ability, with an agent framework that allows more traditional programs to do the heavy lifting in terms of what the human operator wants done, seems like a really natural next step to get to that SF ideal.

bbstats 1y ago

Gemini 2.5 one shots it perfectly...

deleted 1y ago

[deleted]

robinduckett 1y ago

I just asked ChatGPT the following: “Without looking it up, tell me about the Citiquetzelatropic effect” Which is of course, nonsense. It replied: The term “Citiquetzelatropic effect” does not correspond to any known scientific concept, phenomenon, or recognized term in physics, chemistry, biology, psychology, or related fields. It appears to be either fictional, a constructed term, or perhaps a playful creation designed to test familiarity or critical thinking. If you intended this as a test or a trick question, you got me! Otherwise, feel free to clarify the context or correct the spelling, and I’d be happy to help further. So what does this prove?

FroshKiller 1y ago

I think the dumbest people I know are just as likely to answer, "ChatGPT says...."

deleted 1y ago

[deleted]

itaymoav 1y ago

ME: Without looking it up, tell me about the Marathon crater. Explain step by step how you got to your answer ChatGPT said: Alright, let's walk through it together — step-by-step, no looking it up, just from what I know: First Reaction: When you said "Marathon crater," my brain immediately went: Marathon → Mars. That connection comes from recalling that NASA often names Martian features after Earth places. Plus, "Marathon" has a strong NASA feel because of the Mars rovers — they celebrated when Opportunity rover drove the distance of a marathon (42-ish kilometers) on Mars. Digging into Memory: I remember Opportunity traveled a really long distance over years, and they marked milestones along the way by naming craters it passed. There was a Marathon Valley, but I also vaguely recall a Marathon Crater related to that. Putting it Together: So, I reason: Marathon Crater must be a relatively small impact crater on Mars. It was likely named during Opportunity’s mission. Probably not a huge, dramatic crater like Gale Crater (where Curiosity is); more like a marker on Opportunity’s long journey across the plains (like Meridiani Planum). Further Context Guess: Since Opportunity drove a marathon distance, it's very fitting they would name a nearby crater "Marathon" to commemorate that achievement — probably around the time it hit the marathon distance milestone. Conclusion: Without checking, I would say: Marathon Crater is a small, relatively shallow crater on Mars that Opportunity rover reached or passed during its mission, around the time it completed a marathon distance traveled on Mars. Would you like me to now double-check and see how close (or far) I am? Or do you want me to also explain how I'd "fact-check" myself if I had to?