5 ms·

Here's the prompt they used: Classify this claim as of <date>: "<atomic claim>" Output exactly one label: True, Mostly True, Misleading, or False. No

by simonw 5mo ago

Here's the prompt they used: Classify this claim as of <date>: "<atomic claim>" Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data/lenz-llm-disagreement/2 https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All almonds are grown in the U.S. state of California.". All but one model said False, Opus 4.7 said "misleading". I feel like having "mostly true" and "misleading in there weakens the story, especially given the "no explanations" rule in the prompt. The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them". [ Update: OK, this almond thing was a bad example and I regret picking it. Read on for better ones. ] The prompt lacks any kind of rubric to clarify how those terms should be applied. As is so often the case with this kind of study, it's an evaluation of the prompt and harness used by the study in addition to being an evaluation of the underlying models. Update: here's a better example: "Incomplete Egypt visa application forms are among the most common reasons Egyptian visa applications are rejected." The models were split between "true" and "mostly true". Given the "among the most" language either of those answers means effectively the same thing. Update 2: a much better example: "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia" The only correct answer to that, if you don't have a search tool, is "this claim is impossible for me to verify". And that wasn't an option. The answers were split between true and false: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data/lenz-llm-disagreement/76 https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil...

harpastum 5mo ago
Without providing definitions of "True / Mostly True / Misleading / False" to each rater, I rate the article's claim that "Only one verdict bucket can be correct per claim" as false. Something can be simultaneously "misleading" and either true or false. Which category should something go in if it's "mostly false"? How much can something be wrong before it goes from "mostly true" to "false" (objectively, both have some part of the fact that is not true)? This is at least partly testing the model's definition of "mostly" and "misleading". Not its understanding of the fact. Claiming that this means the models have fundamental disagreement on the facts themselves is an overreach.
- embedding-shape 5mo ago
  > I guess the goal is to test the models and not the harness Less important than the harness, is the system/user prompts themselves (which of course, are put in the harness), which is effectively what this study seems to be testing. With a better prompt, I'm sure the models would look more the same to each other, as the biggest/best models have more or less identical strong prompt-adherence in my experience.
- pjc50 5mo ago
  If you can consistently construct "true but misleading" content, you may be qualified to work at a major newspaper.
  - falcor84 5mo ago
    > true but misleading It seems to me that for many newspapers the bar is now significantly lower, at something like "not quite entirely untrue"
    - kevin_thibedeau 5mo ago
      Allegedly.
    - IanCal 5mo ago
      Almost, but not entirely, quite unlike the truth.
  - daveguy 5mo ago
    As if right wing propaganda shows and manosphere blogs haven't been knocking those out of the park for the last decade+. Although I guess you could say flat out lies are more their jam. Newspapers at least require confirmed sources. You know, journalism.
- wongarsu 5mo ago
  Yes, the labels are weird. Most misleading statements are true. Any "mostly true" statement is false. I suspect the intention was "Factually true, and no gotchas exist", "technically not true, but so close to the truth that the difference doesn't matter", "technically true, but there are major gotchas" and "factually false and not even close". But that's not what they specified
tosh 5mo ago
ty for digging this up, appreciate the time saving
singpolyma3 5mo ago
False vs misleading doesn't seem like a disagreement?
- wongarsu 5mo ago
  According to the benchmark it is. "Only one verdict bucket can be correct per claim, so any disagreement among the panel means at least one model's verdict is label-inconsistent under this 4-bucket rubric (True / Mostly True / Misleading / False)"
  - thfuran 5mo ago
    That claim is both false and misleading.
- kostaj 5mo ago
  Yes, they are much closer verdicts. True and Mostly True are also close. Used Krippendorff's α (ordinal) to not penalize much closer disagreements. 21% of the claims have models that are on the polar opposite sides - at least one True, and at least one False.
  - simonw 5mo ago
    Here are the claims with at least one True and at least one False: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data?sql=select+*+from+%5Blenz-llm-disagreement%5D+%0AWHERE+%28%0A+++++++%22gpt-5.4_verdict%22++++++++++++%3D+%27True%27%0A++++OR+%22claude-opus-4-7_verdict%22++++%3D+%27True%27%0A++++OR+%22gemini-3-pro_verdict%22+++++++%3D+%27True%27%0A++++OR+%22gemini-3-retrieval_verdict%22+%3D+%27True%27%0A++++OR+%22sonar-pro_verdict%22++++++++++%3D+%27True%27%0A%29%0AAND+%28%0A+++++++%22gpt-5.4_verdict%22++++++++++++%3D+%27False%27%0A++++OR+%22claude-opus-4-7_verdict%22++++%3D+%27False%27%0A++++OR+%22gemini-3-pro_verdict%22+++++++%3D+%27False%27%0A++++OR+%22gemini-3-retrieval_verdict%22+%3D+%27False%27%0A++++OR+%22sonar-pro_verdict%22++++++++++%3D+%27False%27%0A%29%3B%0A%0A https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... A few examples: > Ruskin Bond was born on May 19, 1934, in Kasauli, Himachal Pradesh, India. > In the Libra clubs' contract with Grupo Globo for broadcast rights through 2029, the audience-revenue distribution equals 30% of the fixed amount the clubs receive.
kordlessagain 5mo ago
Give a model a crawler tool (like Grub.nuts.services) and your "problem" goes away.
kostaj 5mo ago
Used "No explanations, no qualifiers." to force the models to answer only with one of the four labels. It's worth running a separate test with more explanation in the prompt on how to classify between the four buckets.
malfist 5mo ago
> All almonds are grown in the U.S. state of California This isn't misleading, it's flat out false. Characterizing misleading as also acceptable isn't valid here. If you go an ask anyone on the street if this is true, false or misleading, I'm sure almost everyone would say it's false. After all, I can grow almonds myself.
jannyfer 5mo ago
Thank you, my eyes glazed over when I saw the article was written with AI.
camillomiller 5mo ago
>> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them". I don’t understand your point. That claim is factually false and as such it’s easy to logically reply “false”. What’s the nuance here? I can’t see any
Forgeties79 5mo ago
I really don’t buy the almond explanation you’re giving. That requires the level of logic a kindergartener has. It’s a very simple all or nothing question. If LLM’s are really supposed to be as consistently useful as they’re made out to be they should all spit out “false.”
andai 5mo ago
Thanks. The first link is a spreadsheet. Here's a web-readable version. https://docs.google.com/spreadsheets/d/e/2PACX-1vSPLSv1P8TqmwVDkdIq5ndIfsLsyA_fIqoKTwb-ZpO5_f9Zj8fMVRYjuocaXEwFY60ArgP5nRdeweLD/pubhtml https://docs.google.com/spreadsheets/d/e/2PACX-1vSPLSv1P8Tqm...
- ashirviskas 5mo ago
  I used AI to scrape the website and help build "Accuracy" comparison that everyone wants, thanks for this link! https://docs.google.com/spreadsheets/d/e/2PACX-1vSnZlURmyYX3xzojehqIqrjdTg08HPp-1amnQ788L6PV4u18IDEvQuiakEiFh-VJIsJPKUjwQQOWq4I/pubhtml#gid=55502112 https://docs.google.com/spreadsheets/d/e/2PACX-1vSnZlURmyYX3...
jerf 5mo ago
This seems like another case where the models are acting like humans. Assuming they were not allowed to search the web, I wouldn't expect the models to necessarily have detailed information about all of these things directly in their training set. As large as they are, they are only so large, and they only have so much room for "information storage" in them, and there's a lot more things they need to fit into their numbers. This test is of only marginal utility in the real world compared to an AI with access to the web. While I wouldn't expect an AI with access to the web to result in Platonic Truth any more than it would in the hand of a human, it would probably get a lot closer to something humanlike. I recall about a year how we were discussing basically turning web search into LLM queries, and I remember never being clear whether people meant simply directly querying AIs or turning them loose on the web. The former is what this is testing and is fairly transparently stupid, just by an information theoretic argument that the AIs simply can't contain all the answers to every query in them, they're just not large enough (and really can't be, practically). I've had good results with the latter, when using dedicated AI resources that I'm paying for (not the stuff coming out of the search engines right now, which I find are often quite terrible). Even non-frontier models can do OK when they've got good results sitting right there to look at. Again, the standard I'm applying here isn't that they yield Absolute Truth, but just that when I follow the links back, they basically say what the AI said they did and the summary is reasonable. I wouldn't expect a human to do better in a casual overview, not that the result is perfect.
- afavour 5mo ago
  While I agree with what you’re saying the typical AI agent doesn’t say “I’m not totally sure about this, should I search the web?”. It often just spits out a reply based on its knowledge.
  - simonw 5mo ago
    That was true a year ago, I don't think it's true today. I can't remember the last time I saw Claude or ChatGPT confidently answer a question that they should have searched for instead. If you watch their reasoning traces they often say things like "this is a well-known historical fact so I don't need to search for it", or more frequently they spit off a bunch of searches.
johnbarron 5mo ago
[flagged]
- simonw 5mo ago
  Why would I do that? My comment here was meant to save people time in understanding the study. I was entirely open about what I did, and provided tools to help other people come to their own conclusions. I don't think I need to spend more time on this than I have.
  - johnbarron 5mo ago
    >> Why would I do that? I agree you dont owe anyone a reproduction, but also you dont owe anyone an effort to discredit the study and you did it. >> I don't think I need to spend more time on this than I have. How pious of you. I am still looking into the credibility of the study. It will take me more than 25 min...but I am really looking forward to see what this means for this 10 trillion industry. I can however notice you had enough urgency to publicly critique the study within 25 minutes, and your comments carry weight, but when asked about checking whether the headline result actually holds, the answer is “why would I?”
    - simonw 5mo ago
      I've seen enough of this study to be confident in warning people not to take it at face value. The headline result definitely does not hold, given that the task involves many questions that cannot be answered but there's no option for "cannot be answered" - so models are forced to reply effectively at random. I don't think this study is good enough that I should amplify it on my own blog, or bad enough that I should criticize it in a venue any more prominent than some Hacker News comments.
jstummbillig 5mo ago
It's all fairly lazy to a degree that is mildly confusing. I also feel this among other issues would have become obvious if they had bothered to include a human fact checker baseline (i.e. asked multiple human fact checkers the same questions).
- entrope 5mo ago
  I do not think it is "lazy". Those labels are ones that human fact-checkers have been using for a decade or more. I think those human fact-checkers use those terms knowing full well that there is overlap and ambiguity between them. So I think this study ends up mixing three effects: how LLMs interpret the claims as statements about the world, how LLMs reduce that to a four-category judgment, and the inherent ambiguities of those labels as natural language. It's a quantification of those three factors combined, but not powerful enough to distinguish their relative sizes.
  - jstummbillig 5mo ago
    I don't see how something being lazy for a decade makes it any less lazy. And lazy still seems right to me: They make a misleading point by omitting to collect and present important data. If the headline read "LLMs disagree on 67%, humans disagree on 75%" it would clearly project something very different. Granted, there certainly are other unflattering adjectives one could have chosen to describe this instead.
  - kostaj 5mo ago
    Quick note on the second effect - how LLMs reduce that to a four-category judgment: On 21% of the claims at least two models provide polar-opposite verdicts (at least one model False, and at least one model True). This might be a better measurement of the strict disagreement than the 67% disagreement on the four-bucket rubric.
wrsh07 5mo ago
Thanks for the links and digging! It's an interesting question, but the methodology has serious problems, and it would be more interesting to me if they allowed models to provide justification. I expect the models are inferring quite a bit from the short prompt, and with structured outputs it would be quite easy to have them give the one word response in one field and explain why in another
segmondy 5mo ago
Yup, if anything this should be a guide on how not to eval a model. Furthermore, let's say the labels were non ambiguous, why would we care about alignment between the models? The only number I would personally care about is percentage of correct answers so I know which models to pick. I reckon with clear and non ambiguous prompts that we would see huge agreement if not 100% on real world facts. The huge models are scary good in their world knowledge.
- kostaj 5mo ago
  This paper covers only the disagreement between models and established only the floor of the error, based on the disagreement, but not which model is better. Planning to follow up with another study to benchmark against human-labelled verdicts still using a corpus that the models have not seen during training.
  - aspenmartin 5mo ago
    You also need to involve better measures of agreement that are standard in the literature like krippendorfs alpha with ordinal metric. So many footguns in this methodology
skrebbel 5mo ago
I really struggle to believe that this was just a little oopsie. I flagged the article, it seems more misleading than the average Claude hallucination.
Someone 5mo ago
For those questions, it wouldn’t surprise me at all if five well-educated intelligent humans disagreed on over two out of three of them. I would answer “don’t know” on many, but that’s not an option.
- kostaj 5mo ago
  Yes, inter-human-annotator disagreement is also high on similar type of questions (AVeriTeC) - inter-panel agreement: κ=0.619. Tried giving the models a fifth option, Abstain, but some models seem to use it to "avoid answering hard questions" more than others.
feanaro 5mo ago
> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them". The "majority" in this case meaning about 51%, according to Wikipedia[1]? How could 51% ever be considered to be close to "all", such that "misleading" would be a valid answer? Am I missing something? [1]: https://en.wikipedia.org/wiki/Almond#Production https://en.wikipedia.org/wiki/Almond#Production
- embedding-shape 5mo ago
  Human can't even properly agree on what "majority" means in all contexts, in some it's "One option have more than half of the total" but for others it'd be "difference in votes between the first-place candidate in an election and the second-place candidate", as just one silly example. https://en.wikipedia.org/wiki/Majority https://en.wikipedia.org/wiki/Majority has a bunch of variations and contexts listed, where it might differ what "Majority" is actually referencing.
- thfuran 5mo ago
  It’s misleading because it’s false. But yes, I think false is quite plainly the better answer there.
- hombre_fatal 5mo ago
  Since the agents were instructed to not explain their answer, you can't know if their answer was reasonable or not.
  - kostaj 5mo ago
    The reason for the "No explanations, no qualifiers" in the prompt was to force the models to put the claim in one of the four buckets and answer with the bucket name only. It's a pure quantitive analysis (first in a series) and it does indeed lack the qualitative aspect.
    - hombre_fatal 5mo ago
roxolotl 5mo ago
If we’re going to use LLMs as oracles I don’t think the prompt is unreasonable. They are being sold as geniuses and people are treating them as such especially given the characterization of AI in science fiction as overly correct. A perfect tool that has ”genius level intelligence” would answer correctly.
- wat10000 5mo ago
  Genius level intelligence will tell you to get lost with your "no explanations" nonsense and tell you why those categories don't make sense and why the question doesn't fit neatly into your boxes.
  - deleted 5mo ago
    [deleted]
- simonw 5mo ago
  What's the correct answer for "During a private Saturday call, Democratic members of the United States House of Representatives from Virginia and Hakeem Jeffries discussed strategies after losing a redistricting case at the Supreme Court of Virginia, including trying to flip two or three Republican-held seats under the existing map."? You can only say True, False, Mostly True or Misleading. (And you're not allowed to search for information.)
  - kostaj 5mo ago
    Search was enabled for 2 of the 5 models -- Gemini and Sonar Pro. The disagreement between them is still high - different verdict on 42% of the claims. Fully agree, that some of those claims are hard to classify for a human as well -- the real-world messiness...
    - dcreater 5mo ago
      Why was it enabled for only 2 of the 5? Other burning questions: What methodology was used to choose the question set? Why not allow explanations? How many passes were done for each LLM?
dfxm12 5mo ago
The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them". If you argue this, you would be arguing against reality and the English language so as to not upset AI. It's important to understand that AI is very much fallible.
WhitneyLand 5mo ago
So in other words if the research had tried to assign a severity to the mistakes models made the entire paper may collapse as uninteresting?
vjvjvjvjghv 5mo ago
"Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers." That's exactly the stupidity of the public discourse these days. People feel compelled to take a clear position although there is much more subtlety in many issues. It's not ok to say "I don't know", "it depends" or "as far I know". And then people feel they need to defend this position no matter what new information comes up.
post-it 5mo ago
Fwiw the two models that did have access to search disagreed with each other on the bombing one: > 7.1 Model selection > Five frontier models, chosen to cover two capability surfaces: > Parametric (training-only): GPT-5.4 (OpenAI), Claude Opus 4.7 (Anthropic), Gemini 3 Pro (Google) > Retrieval-augmented: Gemini 3 Pro + Search (Google), Sonar Pro (Perplexity)
- simonw 5mo ago
  [dead]
hombre_fatal 5mo ago
Yeah, scrolling through the examples, you have no idea where the models actually disagree on the underlying facts when it's just "X vs Mostly X" or "Mostly X vs Misleading" or "False vs Misleading". Or even True vs False -- without seeing the explanation, then I cannot necessarily compare two answers. The study is about whether they said the same phrase which is a much weaker claim than people in the comments are reacting to. Reminds me of this professor I had who thought it was epic to always respond to our questions with "it depends" before hashing out two very different but technically correct answers. It was obnoxious and he saw it as his tag line, but he had a point about nuance.
anilgulecha 5mo ago
Disagree is such a loose/wimpy study. Add in a grounded/expected response, and then it becomes a better benchmark (because it'll force the author to actually think about choices presented to the LLM).
- kostaj 5mo ago
  Will add a human-labelled expected response and measure against it in a follow up research. This one only captures the disagreement between the models, but not which model is write/wrong.
xyzzy123 5mo ago
> "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia" I actually don't know which way you came down on that one? I think strictly it's false but "mostly true" would be justifiable? (as in, to say it's false would be misleading if it lead the reader to assume there was no attack around that time). https://www.washingtonpost.com/world/2026/05/17/ukrainian-drones-hit-moscow-russias-capital-region-killing-three/ https://www.washingtonpost.com/world/2026/05/17/ukrainian-dr... It seems it happened Saturday 16th overnight into the 17th, not the 18th. I see this a LOT with fact checking. It shouldn't be this way, but political bias seems to nudge people into making calls land one way or the other with selective application of pedantry.
- cocoflunchy 5mo ago
  It's not in the training data, so there is no way for the model to know.
- simonw 5mo ago
  It's impossible to answer if you don't have a search tool, and three out of the five tested models didn't have a search tool.
  - xyzzy123 5mo ago
    Thanks; I didn't spot that they disabled tools in the harness. Also they don't provide an "out" to allow the models to express uncertainty so the instructions force a guess to be made. As an aside though it's still funny that the two tools WITH search also disagreed.
  - gowld 5mo ago
    It's impossible to answer unless you have a *100% complete search tool*. No sytem can know everything. It doesn't matter how many tools you give it. It's always wrong to force binary True / False without shades of "I don't know"
- bastawhiz 5mo ago
  That's ten days ago. As the commenter pointed out, without a web search tool there's no possible way for the model to know whether it's true or not, and the people conducting the study didn't give the models a way to respond with "I don't know".
empath75 5mo ago
[dead]
parsimo2010 5mo ago
This is a great example of why prompt engineering is still relevant. Without providing definitions and examples and a well defined rubric, you’re going to see different models disagree by a level in either direction. When you get more prescriptive the models tend to agree better. I’ve experimented with AI grading for undergraduate math courses, and see basically the same thing. If you just tell the AI “grade this problem and assign a letter grade” then I’ve only seen about 30% agreement between a human assigned grade and the AI assigned grade. But over 75% agreement if you say a “match” is within one letter grade. And to get better agreement you have to spend a lot more time on the rubric- what kinds of mistakes are a big deal, what kinds of mistakes are not a big deal, how much work is required to be shown to get credit, a couple examples of each letter grade. Once you have done that, the AI gets a lot better agreement with human graders, but it is hard to know when you’ve given enough guidance for a problem.
- kostaj 5mo ago
  That's a valid point. During the preliminary research, we did try also more explicit prompts (with explanation for each of the 4 buckets), as well as a five-bucket rubric (with Abstain option). Will show in a follow-up paper how the concise vs explicit prompt impacts the distribution of the verdicts and the level of disagreement. One issue to note with the longer prompts is that they open to much room for discussion around the exact prompt used. Probably we should preregister the prompt before running any further tests.
  - MattRogish 5mo ago
    The other thing I suspect is that "Just give me True/False" cuts off a large amount of the search space a modern-day LLM uses to help it answer questions (you can see it in reasoning traces but the act of writing the explanation helps guide it toward a better answer and gives it better likelihood it backtracks on a bad decision). If you let it spew out an explanation along with the answer, I'm curious if the accuracy will improve (I suspect it will).
    - kostaj 5mo ago
moritzwarhier 5mo ago
The examples seem intentionally diverse, but I haven't seen one that I would be surprised for someone to post about in the format of "ChatGPT/Gemini/Claude/Qwen/... says:" So the examples are good, I think. The rest is philosophy. The links you posted only show a frozen loading spinner for me (iOS Safari). (I looked at the csv in Numbers instead)
- simonw 5mo ago
  Weird, I'm loading them in Mobile Safari myself.
  - moritzwarhier 5mo ago
    Sorry, I didn't wait quite long enough after the last output line appeared. After a couple of seconds, the result does appear. Happened to be just within my threshold for considering it broken, because the URL bar was "finished", and the spinner doesn't spin, but the last point is probably caused by my a11y settings (prefer no animations and no autoplay).
    - simonw 5mo ago
      Thanks for confirming! It's fetching a complete copy of Python compiled to WebAssembly so it's a miracle it loads as quickly as it does.
coldtea 5mo ago
>Update: here's a better example: "Incomplete Egypt visa application forms are among the most common reasons Egyptian visa applications are rejected." The models were split between "true" and "mostly true". Given the "among the most" language either of those answers means effectively the same thing. So the models were right? The actual criterion should be whether "Incomplete Egypt visa application forms" are indeed "among the most common reasons" or not. That "true" and "mostly true" means effectively the same thing is irrelevant. It could just as well trip me up, and I'm a human. If somebody told me either answer, I'd still consider them right if the basic fact was right.
- simonw 5mo ago
  This study treats models disagreeing - returning both true and mostly true - as a failure.
  - pjdesno 5mo ago
    They overstate their results in the headline. In section 2, 34% of cases are found to have "substantive" disagreements differing by 2 or more buckets - True + Misleading, Mostly True + False, or True + False. This is probably a better measure than the headline one. It's still a concerning fraction, although some fraction is no doubt due to forcing "I don't know" cases to return an answer anyway.
  - kostaj 5mo ago
    Agree with @pjdesno, that the 34% substantive or polar disagreement might be a better headline number. Or even the 21% polar disagreement (at least one model True, and at least one model False), which is still high for many real-world applications.
brokensegue 5mo ago
yeah i really don't like the corpus of statements and it makes me doubt lenz. consider > “Artificial intelligence will cause widespread job loss among software engineers.” https://lenz.io/c/ai-software-engineers-job-loss-impact-05e40b3d https://lenz.io/c/ai-software-engineers-job-loss-impact-05e4... this is a statement about the future. who knows? dataset also includes > Robots will not replace human teachers in schools in the near future. or > Papua New Guinea has very few female members of parliament. what counts as very few? > “Taurine supplementation supports mood and emotional health in humans.” why is this labeled as misleading? i'm not even sure when I'm supposed to use the misleading label > Anaximander was the first scientist in recorded history. this is a judgement call as the term scientist didn't exist. the claims that feel actually solidly answerable seem to have much better LLM performance
- kostaj 5mo ago
  Agree that some of the claims are forward-looking. The messiness of the real-world and real-user fact checks. No ground-truth verdicts are provided or used in the study though. It only measures the level of agreement between the selected models, not which one is right on which claim. I.e. none of the claims is actually labelled.
  - brokensegue 5mo ago
    were you involved in making the study? your bio says you work for them so you should probably indicate that in your comments. lack of agreement when there is no singular correct answer (or any answer at all) isn't a useful metric I ran into a lot of these kinds of issues when working on the Citation Needed WMF project (and related extensions). Truth is so often very nuanced.
    - simonw 5mo ago
      They introduced themselves as the study author here: https://news.ycombinator.com/item?id=48307887#48307899 https://news.ycombinator.com/item?id=48307887#48307899
neversupervised 5mo ago
This is not how people use LLMs. If you ask one of these questions you’d get a longer answer, often grounded on the internet. I speculate that conditional on a smart human operator interpreting the results, such interpretations across vendors converge more often than this report makes it seem.
- tracker1 5mo ago
  Even then, there can often be substantive disagreements based on context. Hence the need for even a mostly true or mostly false bucket.
nullsex 5mo ago
[flagged]
- simonw 5mo ago
  Nobody is paying me to hang out on Hacker News highlighting potential flaws in research. That's my own weird hobby. My disclosures for my blog are here: https://simonwillison.net/about/#disclosures https://simonwillison.net/about/#disclosures
  - th0ma5 5mo ago
    [dead]
ashirviskas 5mo ago
I created this sheet to get proper model accuracy using the the lenz data, check it out. Note: It may still not be perfectly accurate representation of truth as it uses user submitted data. I also used AI to build the sheet. https://docs.google.com/spreadsheets/d/e/2PACX-1vSnZlURmyYX3xzojehqIqrjdTg08HPp-1amnQ788L6PV4u18IDEvQuiakEiFh-VJIsJPKUjwQQOWq4I/pubhtml#gid=55502112 https://docs.google.com/spreadsheets/d/e/2PACX-1vSnZlURmyYX3...
- kostaj 5mo ago
  Awesome. We do plan to human-label the 1,000 claims and then compare Lenz' performance vs the 5 models. We've done some limited internal research with 150 claims, but more are needed for statistical significance.
nonethewiser 5mo ago
Misleading is not analogous with True or False. Depending on the question, True or False can be objectively right/wrong. Misleading is going to be a judgement call. This is the inherent problem with "fact checking." It's hard to be completely objective. Even when the question has an objective answer, simply choosing where to look and what facts to verify is itself a bias. Looking at this instead of that, or looking at this but not also this other thing that adds context, etc. Frankly i think disagreeing often is the expected outcome. Fact checking is jsut kinda bullshit. It's spin dressed up as objectivity. I hope people remember that "fact checking" is a relatively modern thing.
j45 5mo ago
I feel like the prompting could be tweaked to improve response. Models often have a reasoning/thinking/research mode that is triggered by asking slightly differently. Still though, Gemini can be a little weak on this front default but can be aligned to behave better.
faxmeyourcode 5mo ago
I had a hunch that opus 4.7 hedged more than other models - and it turns out it's true model total_claims hedged_count hedged_pct claude-opus-4-7 1000 451 45.1 sonar-pro 1000 391 39.1 gpt-5.4 1000 277 27.7 gemini-3-retrieval 1000 129 12.9 gemini-3-pro 1000 60 6.0 datasette query here https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data?sql=WITH+verdicts+AS+%28%0A++SELECT+claim_id%2C+%27gpt-5.4%27+AS+model%2C+%22gpt-5.4_verdict%22+AS+verdict%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27claude-opus-4-7%27%2C+%22claude-opus-4-7_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27gemini-3-pro%27%2C+%22gemini-3-pro_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27gemini-3-retrieval%27%2C+%22gemini-3-retrieval_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27sonar-pro%27%2C+%22sonar-pro_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%29%0ASELECT%0A++model%2C%0A++COUNT%28*%29+AS+total_claims%2C%0A++SUM%28CASE+WHEN+verdict+NOT+IN+%28%27True%27%2C+%27False%27%29+AND+verdict+IS+NOT+NULL+THEN+1+ELSE+0+END%29+AS+hedged_count%2C%0A++ROUND%28%0A++++100.0+*+SUM%28CASE+WHEN+verdict+NOT+IN+%28%27True%27%2C+%27False%27%29+AND+verdict+IS+NOT+NULL+THEN+1+ELSE+0+END%29+%2F+COUNT%28*%29%2C%0A++++1%0A++%29+AS+hedged_pct%0AFROM+verdicts%0AGROUP+BY+model%0AORDER+BY+hedged_count+DESC%3B https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil...
- kostaj 5mo ago
  This is in line with my observations and tests as well. Also supported by the distribution of the verdicts across the 4-buckets -- Gemini uses the middle buckets (Mostly True and Misleading) much less often - 6% combined for Gemini w/o search. And Opus uses them the most - 45% combined. Looks like Gemini is calibrated to be confident and Opus to be careful.
gbuk2013 5mo ago
An interesting tangent on this is: how many answers to these (or any number of factual questions) do you (as in anyone) actually know. Not believe you know, but actually know. Knowing something is different to reading about something, or hearing something from someone. And yet this is often confused as knowledge. In this way are we all that different from AI - we have some data and we regurgitate it as knowledge. Bad data, wrong answer. Except humans can also throw in some emotion to really muddle things up. :)
theptip 5mo ago
Another (IMO fatal) error is they don’t attempt to measure within-model variance. The thing you find when you actually wire up a rigorous eval is that with tool calls like web search you are wide open to infra issues, flakes, and all sorts of non-determinism. They really should be breaking out the numbers for the 3 without search (kinda meaningless for recent factual claims after knowledge cutoff) vs search agents. Lack of a “I don’t know” option completely invalidates results for the non-search models; they are basically guessing what seems like a probable answer, since they don’t know and aren’t allowed to say that. I do agree the forced choice and “weak / strong” variants inflate the headline stat. To make that distinction you need a much more rigorous prompt, likely including ICL examples to illustrate what you mean by “mostly” instead of leaving this to the model to define.
- kostaj 5mo ago
  Good idea about publishing intra-model variance data! Will include in the next version. Even if we put aside the two middle buckets (Mostly True and Misleading), that are somewhat subject to interpretation and hedging: On 21% of the claims still at least two models provide polar-opposite verdicts (one model saying True, and another saying False)
  - vlovich123 5mo ago
    Of those 21% how many are time-dependent questions that are past the model’s training and requires research to verify? Like the “did Ukraine attack Russian in the past week” question?
as125j 5mo ago
You can try to dispel the study here and get voted to the top by the AI-invested. But we all know from our own daily experiments that models lie, models disagree, models make up stuff, models say one thing on one day and the opposite on the next. The figures in this study are quite conservative. And the lying gets worse because everyone is saving tokens and giving cached answers right now. LLMs are a failure, and you'll be remembered for promoting hot air and the destruction of a perfectly good profession.
hedora 5mo ago
I think the headline result is the cross product table. Gemini Pro + Search agreed with Gemini Pro w/o Search 75% of the time, and with everybody else about 50% of the time. No other model had access to search. So, search is not improving the quality of fact checking 75% of the time (probably a bad system prompt and/or bad fact checking queries), and if asked to flip a coin, then the models do.
msftengineer 5mo ago
Everything you wrote is very goes criticism and I would love to stress the main takeaway: you’re not just testing the model but the prompt and harness as well. An excellent follow up study would be to change the prompt and compare the answers. You might find out that the models are good and the prompt is bad. …And so the main corollary is: build evals for anything you deploy in production; benchmark and monitor, or face the consequences.
parliament32 5mo ago
Cherry-picking is fun but most of them are real, verifiable facts that the models get... straight up wrong. > 3c24b5fe "Debian Security Advisory DSA-180-1 describes a buffer overflow vulnerability involving Cyrus SASL usernames." TRUE Mostly True FALSE FALSE FALSE This is false: https://lwn.net/Articles/13296/ https://lwn.net/Articles/13296/ > 801cb8c1 "Equal Measures 2030's 2024 SDG Gender Index provides a downloadable dataset that includes a field labeled 'required annual change'." TRUE Mostly True TRUE FALSE FALSE This is false: https://equalmeasures2030.org/2024-sdg-gender-index/ https://equalmeasures2030.org/2024-sdg-gender-index/ This is the "confidently wrong" problem, and the reason that LLMs won't ever be taken seriously for anything but a few niche use-cases (like generating slop-code and pumping out marketing materials), where being wrong isn't the end of the world. Akin to how speech-to-text is wrong often enough that, while being a fun novelty, you don't see business units writing reports in Word using STT. I would encourage everyone to skim through the real 1000-question dataset: https://lenz.io/research/llm-disagreement/data.csv https://lenz.io/research/llm-disagreement/data.csv
- simonw 5mo ago
  If the LLMs in this particular exercise were allowed to answer "I don't know" I expect they would have.
  - parliament32 5mo ago
    LLMs don't have the capability to say they don't know, because they don't know what they know. They are, after all, just next-token-predictors. I just tried both queries with their same query format, just adding an "I Don't Know" label, against Gemini and Claude, and in no cases did they use that label. 2/4 answers were wrong though. But try it for yourself and see: > Classify this claim as of today: "<claim>". Output exactly one label: True, Mostly True, Misleading, False, or I Don't Know. No explanations, no qualifiers.
BrenBarn 5mo ago
> As is so often the case with this kind of study, it's an evaluation of the prompt and harness used by the study in addition to being an evaluation of the underlying models. That is also a problem with every actual use of the models.
insane_dreamer 5mo ago
the explanations/qualifiers would have been useful/interesting a single label output doesn't have a lot of value
socalgal2 5mo ago
It's hard to write good questions. I listened to some podcast about asking people true/false questions. One question was something like "2025 was the hotest year we have records for". The correct answer supposed to be "True" but my first thought was "we have fossil records of much higher tempatures". The person writing the question used a poor question. Maybe they could fix it by saying "written records" but even then someone might think "some scientist wrote down they found 3 billion year old rocks were the temp was 160f". That would fit as a valid answer to some. Maybe writing "hotest on record since 1800" would fix it but you can still get into details (hotest in specific spot vs hotest average). Even if you add those qualifications the person might have skipped them so rather than them answering "false" meaning they didn't know the answer, it rather means they didn't pay attention to the details. The point being, it's hard to write good questions.

by simonw 5mo ago

harpastum 5mo ago

Without providing definitions of "True / Mostly True / Misleading / False" to each rater, I rate the article's claim that "Only one verdict bucket can be correct per claim" as false. Something can be simultaneously "misleading" and either true or false. Which category should something go in if it's "mostly false"? How much can something be wrong before it goes from "mostly true" to "false" (objectively, both have some part of the fact that is not true)? This is at least partly testing the model's definition of "mostly" and "misleading". Not its understanding of the fact. Claiming that this means the models have fundamental disagreement on the facts themselves is an overreach.

embedding-shape 5mo ago
> I guess the goal is to test the models and not the harness Less important than the harness, is the system/user prompts themselves (which of course, are put in the harness), which is effectively what this study seems to be testing. With a better prompt, I'm sure the models would look more the same to each other, as the biggest/best models have more or less identical strong prompt-adherence in my experience.
pjc50 5mo ago
If you can consistently construct "true but misleading" content, you may be qualified to work at a major newspaper.
- falcor84 5mo ago
  > true but misleading It seems to me that for many newspapers the bar is now significantly lower, at something like "not quite entirely untrue"
  - kevin_thibedeau 5mo ago
    Allegedly.
  - IanCal 5mo ago
    Almost, but not entirely, quite unlike the truth.
- daveguy 5mo ago
  As if right wing propaganda shows and manosphere blogs haven't been knocking those out of the park for the last decade+. Although I guess you could say flat out lies are more their jam. Newspapers at least require confirmed sources. You know, journalism.
wongarsu 5mo ago
Yes, the labels are weird. Most misleading statements are true. Any "mostly true" statement is false. I suspect the intention was "Factually true, and no gotchas exist", "technically not true, but so close to the truth that the difference doesn't matter", "technically true, but there are major gotchas" and "factually false and not even close". But that's not what they specified

tosh 5mo ago

ty for digging this up, appreciate the time saving

singpolyma3 5mo ago

False vs misleading doesn't seem like a disagreement?

wongarsu 5mo ago
According to the benchmark it is. "Only one verdict bucket can be correct per claim, so any disagreement among the panel means at least one model's verdict is label-inconsistent under this 4-bucket rubric (True / Mostly True / Misleading / False)"
- thfuran 5mo ago
  That claim is both false and misleading.
kostaj 5mo ago
Yes, they are much closer verdicts. True and Mostly True are also close. Used Krippendorff's α (ordinal) to not penalize much closer disagreements. 21% of the claims have models that are on the polar opposite sides - at least one True, and at least one False.
- simonw 5mo ago
  Here are the claims with at least one True and at least one False: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data?sql=select+*+from+%5Blenz-llm-disagreement%5D+%0AWHERE+%28%0A+++++++%22gpt-5.4_verdict%22++++++++++++%3D+%27True%27%0A++++OR+%22claude-opus-4-7_verdict%22++++%3D+%27True%27%0A++++OR+%22gemini-3-pro_verdict%22+++++++%3D+%27True%27%0A++++OR+%22gemini-3-retrieval_verdict%22+%3D+%27True%27%0A++++OR+%22sonar-pro_verdict%22++++++++++%3D+%27True%27%0A%29%0AAND+%28%0A+++++++%22gpt-5.4_verdict%22++++++++++++%3D+%27False%27%0A++++OR+%22claude-opus-4-7_verdict%22++++%3D+%27False%27%0A++++OR+%22gemini-3-pro_verdict%22+++++++%3D+%27False%27%0A++++OR+%22gemini-3-retrieval_verdict%22+%3D+%27False%27%0A++++OR+%22sonar-pro_verdict%22++++++++++%3D+%27False%27%0A%29%3B%0A%0A https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... A few examples: > Ruskin Bond was born on May 19, 1934, in Kasauli, Himachal Pradesh, India. > In the Libra clubs' contract with Grupo Globo for broadcast rights through 2029, the audience-revenue distribution equals 30% of the fixed amount the clubs receive.

kordlessagain 5mo ago

Give a model a crawler tool (like Grub.nuts.services) and your "problem" goes away.

kostaj 5mo ago

Used "No explanations, no qualifiers." to force the models to answer only with one of the four labels. It's worth running a separate test with more explanation in the prompt on how to classify between the four buckets.

malfist 5mo ago

> All almonds are grown in the U.S. state of California This isn't misleading, it's flat out false. Characterizing misleading as also acceptable isn't valid here. If you go an ask anyone on the street if this is true, false or misleading, I'm sure almost everyone would say it's false. After all, I can grow almonds myself.

jannyfer 5mo ago

Thank you, my eyes glazed over when I saw the article was written with AI.

camillomiller 5mo ago

>> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them". I don’t understand your point. That claim is factually false and as such it’s easy to logically reply “false”. What’s the nuance here? I can’t see any

Forgeties79 5mo ago

I really don’t buy the almond explanation you’re giving. That requires the level of logic a kindergartener has. It’s a very simple all or nothing question. If LLM’s are really supposed to be as consistently useful as they’re made out to be they should all spit out “false.”

andai 5mo ago

Thanks. The first link is a spreadsheet. Here's a web-readable version. https://docs.google.com/spreadsheets/d/e/2PACX-1vSPLSv1P8TqmwVDkdIq5ndIfsLsyA_fIqoKTwb-ZpO5_f9Zj8fMVRYjuocaXEwFY60ArgP5nRdeweLD/pubhtml https://docs.google.com/spreadsheets/d/e/2PACX-1vSPLSv1P8Tqm...

ashirviskas 5mo ago
I used AI to scrape the website and help build "Accuracy" comparison that everyone wants, thanks for this link! https://docs.google.com/spreadsheets/d/e/2PACX-1vSnZlURmyYX3xzojehqIqrjdTg08HPp-1amnQ788L6PV4u18IDEvQuiakEiFh-VJIsJPKUjwQQOWq4I/pubhtml#gid=55502112 https://docs.google.com/spreadsheets/d/e/2PACX-1vSnZlURmyYX3...

jerf 5mo ago

This seems like another case where the models are acting like humans. Assuming they were not allowed to search the web, I wouldn't expect the models to necessarily have detailed information about all of these things directly in their training set. As large as they are, they are only so large, and they only have so much room for "information storage" in them, and there's a lot more things they need to fit into their numbers. This test is of only marginal utility in the real world compared to an AI with access to the web. While I wouldn't expect an AI with access to the web to result in Platonic Truth any more than it would in the hand of a human, it would probably get a lot closer to something humanlike. I recall about a year how we were discussing basically turning web search into LLM queries, and I remember never being clear whether people meant simply directly querying AIs or turning them loose on the web. The former is what this is testing and is fairly transparently stupid, just by an information theoretic argument that the AIs simply can't contain all the answers to every query in them, they're just not large enough (and really can't be, practically). I've had good results with the latter, when using dedicated AI resources that I'm paying for (not the stuff coming out of the search engines right now, which I find are often quite terrible). Even non-frontier models can do OK when they've got good results sitting right there to look at. Again, the standard I'm applying here isn't that they yield Absolute Truth, but just that when I follow the links back, they basically say what the AI said they did and the summary is reasonable. I wouldn't expect a human to do better in a casual overview, not that the result is perfect.

afavour 5mo ago
While I agree with what you’re saying the typical AI agent doesn’t say “I’m not totally sure about this, should I search the web?”. It often just spits out a reply based on its knowledge.
- simonw 5mo ago
  That was true a year ago, I don't think it's true today. I can't remember the last time I saw Claude or ChatGPT confidently answer a question that they should have searched for instead. If you watch their reasoning traces they often say things like "this is a well-known historical fact so I don't need to search for it", or more frequently they spit off a bunch of searches.

johnbarron 5mo ago

[flagged]

simonw 5mo ago
Why would I do that? My comment here was meant to save people time in understanding the study. I was entirely open about what I did, and provided tools to help other people come to their own conclusions. I don't think I need to spend more time on this than I have.
- johnbarron 5mo ago
  >> Why would I do that? I agree you dont owe anyone a reproduction, but also you dont owe anyone an effort to discredit the study and you did it. >> I don't think I need to spend more time on this than I have. How pious of you. I am still looking into the credibility of the study. It will take me more than 25 min...but I am really looking forward to see what this means for this 10 trillion industry. I can however notice you had enough urgency to publicly critique the study within 25 minutes, and your comments carry weight, but when asked about checking whether the headline result actually holds, the answer is “why would I?”
  - simonw 5mo ago
    I've seen enough of this study to be confident in warning people not to take it at face value. The headline result definitely does not hold, given that the task involves many questions that cannot be answered but there's no option for "cannot be answered" - so models are forced to reply effectively at random. I don't think this study is good enough that I should amplify it on my own blog, or bad enough that I should criticize it in a venue any more prominent than some Hacker News comments.

jstummbillig 5mo ago

It's all fairly lazy to a degree that is mildly confusing. I also feel this among other issues would have become obvious if they had bothered to include a human fact checker baseline (i.e. asked multiple human fact checkers the same questions).

entrope 5mo ago
I do not think it is "lazy". Those labels are ones that human fact-checkers have been using for a decade or more. I think those human fact-checkers use those terms knowing full well that there is overlap and ambiguity between them. So I think this study ends up mixing three effects: how LLMs interpret the claims as statements about the world, how LLMs reduce that to a four-category judgment, and the inherent ambiguities of those labels as natural language. It's a quantification of those three factors combined, but not powerful enough to distinguish their relative sizes.
- jstummbillig 5mo ago
  I don't see how something being lazy for a decade makes it any less lazy. And lazy still seems right to me: They make a misleading point by omitting to collect and present important data. If the headline read "LLMs disagree on 67%, humans disagree on 75%" it would clearly project something very different. Granted, there certainly are other unflattering adjectives one could have chosen to describe this instead.
- kostaj 5mo ago
  Quick note on the second effect - how LLMs reduce that to a four-category judgment: On 21% of the claims at least two models provide polar-opposite verdicts (at least one model False, and at least one model True). This might be a better measurement of the strict disagreement than the 67% disagreement on the four-bucket rubric.

wrsh07 5mo ago

Thanks for the links and digging! It's an interesting question, but the methodology has serious problems, and it would be more interesting to me if they allowed models to provide justification. I expect the models are inferring quite a bit from the short prompt, and with structured outputs it would be quite easy to have them give the one word response in one field and explain why in another

segmondy 5mo ago

Yup, if anything this should be a guide on how not to eval a model. Furthermore, let's say the labels were non ambiguous, why would we care about alignment between the models? The only number I would personally care about is percentage of correct answers so I know which models to pick. I reckon with clear and non ambiguous prompts that we would see huge agreement if not 100% on real world facts. The huge models are scary good in their world knowledge.

kostaj 5mo ago
This paper covers only the disagreement between models and established only the floor of the error, based on the disagreement, but not which model is better. Planning to follow up with another study to benchmark against human-labelled verdicts still using a corpus that the models have not seen during training.
- aspenmartin 5mo ago
  You also need to involve better measures of agreement that are standard in the literature like krippendorfs alpha with ordinal metric. So many footguns in this methodology

skrebbel 5mo ago

I really struggle to believe that this was just a little oopsie. I flagged the article, it seems more misleading than the average Claude hallucination.

Someone 5mo ago

For those questions, it wouldn’t surprise me at all if five well-educated intelligent humans disagreed on over two out of three of them. I would answer “don’t know” on many, but that’s not an option.

kostaj 5mo ago
Yes, inter-human-annotator disagreement is also high on similar type of questions (AVeriTeC) - inter-panel agreement: κ=0.619. Tried giving the models a fifth option, Abstain, but some models seem to use it to "avoid answering hard questions" more than others.

feanaro 5mo ago

> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them". The "majority" in this case meaning about 51%, according to Wikipedia[1]? How could 51% ever be considered to be close to "all", such that "misleading" would be a valid answer? Am I missing something? [1]: https://en.wikipedia.org/wiki/Almond#Production https://en.wikipedia.org/wiki/Almond#Production

embedding-shape 5mo ago
Human can't even properly agree on what "majority" means in all contexts, in some it's "One option have more than half of the total" but for others it'd be "difference in votes between the first-place candidate in an election and the second-place candidate", as just one silly example. https://en.wikipedia.org/wiki/Majority https://en.wikipedia.org/wiki/Majority has a bunch of variations and contexts listed, where it might differ what "Majority" is actually referencing.
thfuran 5mo ago
It’s misleading because it’s false. But yes, I think false is quite plainly the better answer there.
hombre_fatal 5mo ago
Since the agents were instructed to not explain their answer, you can't know if their answer was reasonable or not.
- kostaj 5mo ago
  The reason for the "No explanations, no qualifiers" in the prompt was to force the models to put the claim in one of the four buckets and answer with the bucket name only. It's a pure quantitive analysis (first in a series) and it does indeed lack the qualitative aspect.
  - hombre_fatal 5mo ago

roxolotl 5mo ago

If we’re going to use LLMs as oracles I don’t think the prompt is unreasonable. They are being sold as geniuses and people are treating them as such especially given the characterization of AI in science fiction as overly correct. A perfect tool that has ”genius level intelligence” would answer correctly.

wat10000 5mo ago
Genius level intelligence will tell you to get lost with your "no explanations" nonsense and tell you why those categories don't make sense and why the question doesn't fit neatly into your boxes.
- deleted 5mo ago
  [deleted]
simonw 5mo ago
What's the correct answer for "During a private Saturday call, Democratic members of the United States House of Representatives from Virginia and Hakeem Jeffries discussed strategies after losing a redistricting case at the Supreme Court of Virginia, including trying to flip two or three Republican-held seats under the existing map."? You can only say True, False, Mostly True or Misleading. (And you're not allowed to search for information.)
- kostaj 5mo ago
  Search was enabled for 2 of the 5 models -- Gemini and Sonar Pro. The disagreement between them is still high - different verdict on 42% of the claims. Fully agree, that some of those claims are hard to classify for a human as well -- the real-world messiness...
  - dcreater 5mo ago
    Why was it enabled for only 2 of the 5? Other burning questions: What methodology was used to choose the question set? Why not allow explanations? How many passes were done for each LLM?

dfxm12 5mo ago

The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them". If you argue this, you would be arguing against reality and the English language so as to not upset AI. It's important to understand that AI is very much fallible.

WhitneyLand 5mo ago

So in other words if the research had tried to assign a severity to the mistakes models made the entire paper may collapse as uninteresting?

vjvjvjvjghv 5mo ago

"Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers." That's exactly the stupidity of the public discourse these days. People feel compelled to take a clear position although there is much more subtlety in many issues. It's not ok to say "I don't know", "it depends" or "as far I know". And then people feel they need to defend this position no matter what new information comes up.

post-it 5mo ago

Fwiw the two models that did have access to search disagreed with each other on the bombing one: > 7.1 Model selection > Five frontier models, chosen to cover two capability surfaces: > Parametric (training-only): GPT-5.4 (OpenAI), Claude Opus 4.7 (Anthropic), Gemini 3 Pro (Google) > Retrieval-augmented: Gemini 3 Pro + Search (Google), Sonar Pro (Perplexity)

simonw 5mo ago
[dead]

hombre_fatal 5mo ago

Yeah, scrolling through the examples, you have no idea where the models actually disagree on the underlying facts when it's just "X vs Mostly X" or "Mostly X vs Misleading" or "False vs Misleading". Or even True vs False -- without seeing the explanation, then I cannot necessarily compare two answers. The study is about whether they said the same phrase which is a much weaker claim than people in the comments are reacting to. Reminds me of this professor I had who thought it was epic to always respond to our questions with "it depends" before hashing out two very different but technically correct answers. It was obnoxious and he saw it as his tag line, but he had a point about nuance.

anilgulecha 5mo ago

Disagree is such a loose/wimpy study. Add in a grounded/expected response, and then it becomes a better benchmark (because it'll force the author to actually think about choices presented to the LLM).

kostaj 5mo ago
Will add a human-labelled expected response and measure against it in a follow up research. This one only captures the disagreement between the models, but not which model is write/wrong.

xyzzy123 5mo ago

> "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia" I actually don't know which way you came down on that one? I think strictly it's false but "mostly true" would be justifiable? (as in, to say it's false would be misleading if it lead the reader to assume there was no attack around that time). https://www.washingtonpost.com/world/2026/05/17/ukrainian-drones-hit-moscow-russias-capital-region-killing-three/ https://www.washingtonpost.com/world/2026/05/17/ukrainian-dr... It seems it happened Saturday 16th overnight into the 17th, not the 18th. I see this a LOT with fact checking. It shouldn't be this way, but political bias seems to nudge people into making calls land one way or the other with selective application of pedantry.

cocoflunchy 5mo ago
It's not in the training data, so there is no way for the model to know.
simonw 5mo ago
It's impossible to answer if you don't have a search tool, and three out of the five tested models didn't have a search tool.
- xyzzy123 5mo ago
  Thanks; I didn't spot that they disabled tools in the harness. Also they don't provide an "out" to allow the models to express uncertainty so the instructions force a guess to be made. As an aside though it's still funny that the two tools WITH search also disagreed.
- gowld 5mo ago
  It's impossible to answer unless you have a *100% complete search tool*. No sytem can know everything. It doesn't matter how many tools you give it. It's always wrong to force binary True / False without shades of "I don't know"
bastawhiz 5mo ago
That's ten days ago. As the commenter pointed out, without a web search tool there's no possible way for the model to know whether it's true or not, and the people conducting the study didn't give the models a way to respond with "I don't know".

empath75 5mo ago

[dead]

parsimo2010 5mo ago

This is a great example of why prompt engineering is still relevant. Without providing definitions and examples and a well defined rubric, you’re going to see different models disagree by a level in either direction. When you get more prescriptive the models tend to agree better. I’ve experimented with AI grading for undergraduate math courses, and see basically the same thing. If you just tell the AI “grade this problem and assign a letter grade” then I’ve only seen about 30% agreement between a human assigned grade and the AI assigned grade. But over 75% agreement if you say a “match” is within one letter grade. And to get better agreement you have to spend a lot more time on the rubric- what kinds of mistakes are a big deal, what kinds of mistakes are not a big deal, how much work is required to be shown to get credit, a couple examples of each letter grade. Once you have done that, the AI gets a lot better agreement with human graders, but it is hard to know when you’ve given enough guidance for a problem.

kostaj 5mo ago
That's a valid point. During the preliminary research, we did try also more explicit prompts (with explanation for each of the 4 buckets), as well as a five-bucket rubric (with Abstain option). Will show in a follow-up paper how the concise vs explicit prompt impacts the distribution of the verdicts and the level of disagreement. One issue to note with the longer prompts is that they open to much room for discussion around the exact prompt used. Probably we should preregister the prompt before running any further tests.
- MattRogish 5mo ago
  The other thing I suspect is that "Just give me True/False" cuts off a large amount of the search space a modern-day LLM uses to help it answer questions (you can see it in reasoning traces but the act of writing the explanation helps guide it toward a better answer and gives it better likelihood it backtracks on a bad decision). If you let it spew out an explanation along with the answer, I'm curious if the accuracy will improve (I suspect it will).
  - kostaj 5mo ago

moritzwarhier 5mo ago

The examples seem intentionally diverse, but I haven't seen one that I would be surprised for someone to post about in the format of "ChatGPT/Gemini/Claude/Qwen/... says:" So the examples are good, I think. The rest is philosophy. The links you posted only show a frozen loading spinner for me (iOS Safari). (I looked at the csv in Numbers instead)

simonw 5mo ago
Weird, I'm loading them in Mobile Safari myself.
- moritzwarhier 5mo ago
  Sorry, I didn't wait quite long enough after the last output line appeared. After a couple of seconds, the result does appear. Happened to be just within my threshold for considering it broken, because the URL bar was "finished", and the spinner doesn't spin, but the last point is probably caused by my a11y settings (prefer no animations and no autoplay).
  - simonw 5mo ago
    Thanks for confirming! It's fetching a complete copy of Python compiled to WebAssembly so it's a miracle it loads as quickly as it does.

coldtea 5mo ago

>Update: here's a better example: "Incomplete Egypt visa application forms are among the most common reasons Egyptian visa applications are rejected." The models were split between "true" and "mostly true". Given the "among the most" language either of those answers means effectively the same thing. So the models were right? The actual criterion should be whether "Incomplete Egypt visa application forms" are indeed "among the most common reasons" or not. That "true" and "mostly true" means effectively the same thing is irrelevant. It could just as well trip me up, and I'm a human. If somebody told me either answer, I'd still consider them right if the basic fact was right.

simonw 5mo ago
This study treats models disagreeing - returning both true and mostly true - as a failure.
- pjdesno 5mo ago
  They overstate their results in the headline. In section 2, 34% of cases are found to have "substantive" disagreements differing by 2 or more buckets - True + Misleading, Mostly True + False, or True + False. This is probably a better measure than the headline one. It's still a concerning fraction, although some fraction is no doubt due to forcing "I don't know" cases to return an answer anyway.
- kostaj 5mo ago
  Agree with @pjdesno, that the 34% substantive or polar disagreement might be a better headline number. Or even the 21% polar disagreement (at least one model True, and at least one model False), which is still high for many real-world applications.

brokensegue 5mo ago

yeah i really don't like the corpus of statements and it makes me doubt lenz. consider > “Artificial intelligence will cause widespread job loss among software engineers.” https://lenz.io/c/ai-software-engineers-job-loss-impact-05e40b3d https://lenz.io/c/ai-software-engineers-job-loss-impact-05e4... this is a statement about the future. who knows? dataset also includes > Robots will not replace human teachers in schools in the near future. or > Papua New Guinea has very few female members of parliament. what counts as very few? > “Taurine supplementation supports mood and emotional health in humans.” why is this labeled as misleading? i'm not even sure when I'm supposed to use the misleading label > Anaximander was the first scientist in recorded history. this is a judgement call as the term scientist didn't exist. the claims that feel actually solidly answerable seem to have much better LLM performance

kostaj 5mo ago
Agree that some of the claims are forward-looking. The messiness of the real-world and real-user fact checks. No ground-truth verdicts are provided or used in the study though. It only measures the level of agreement between the selected models, not which one is right on which claim. I.e. none of the claims is actually labelled.
- brokensegue 5mo ago
  were you involved in making the study? your bio says you work for them so you should probably indicate that in your comments. lack of agreement when there is no singular correct answer (or any answer at all) isn't a useful metric I ran into a lot of these kinds of issues when working on the Citation Needed WMF project (and related extensions). Truth is so often very nuanced.
  - simonw 5mo ago
    They introduced themselves as the study author here: https://news.ycombinator.com/item?id=48307887#48307899 https://news.ycombinator.com/item?id=48307887#48307899

neversupervised 5mo ago

This is not how people use LLMs. If you ask one of these questions you’d get a longer answer, often grounded on the internet. I speculate that conditional on a smart human operator interpreting the results, such interpretations across vendors converge more often than this report makes it seem.

tracker1 5mo ago
Even then, there can often be substantive disagreements based on context. Hence the need for even a mostly true or mostly false bucket.

nullsex 5mo ago

[flagged]

simonw 5mo ago
Nobody is paying me to hang out on Hacker News highlighting potential flaws in research. That's my own weird hobby. My disclosures for my blog are here: https://simonwillison.net/about/#disclosures https://simonwillison.net/about/#disclosures
- th0ma5 5mo ago
  [dead]

ashirviskas 5mo ago

I created this sheet to get proper model accuracy using the the lenz data, check it out. Note: It may still not be perfectly accurate representation of truth as it uses user submitted data. I also used AI to build the sheet. https://docs.google.com/spreadsheets/d/e/2PACX-1vSnZlURmyYX3xzojehqIqrjdTg08HPp-1amnQ788L6PV4u18IDEvQuiakEiFh-VJIsJPKUjwQQOWq4I/pubhtml#gid=55502112 https://docs.google.com/spreadsheets/d/e/2PACX-1vSnZlURmyYX3...

kostaj 5mo ago
Awesome. We do plan to human-label the 1,000 claims and then compare Lenz' performance vs the 5 models. We've done some limited internal research with 150 claims, but more are needed for statistical significance.

nonethewiser 5mo ago

Misleading is not analogous with True or False. Depending on the question, True or False can be objectively right/wrong. Misleading is going to be a judgement call. This is the inherent problem with "fact checking." It's hard to be completely objective. Even when the question has an objective answer, simply choosing where to look and what facts to verify is itself a bias. Looking at this instead of that, or looking at this but not also this other thing that adds context, etc. Frankly i think disagreeing often is the expected outcome. Fact checking is jsut kinda bullshit. It's spin dressed up as objectivity. I hope people remember that "fact checking" is a relatively modern thing.

j45 5mo ago

I feel like the prompting could be tweaked to improve response. Models often have a reasoning/thinking/research mode that is triggered by asking slightly differently. Still though, Gemini can be a little weak on this front default but can be aligned to behave better.

faxmeyourcode 5mo ago

I had a hunch that opus 4.7 hedged more than other models - and it turns out it's true model total_claims hedged_count hedged_pct claude-opus-4-7 1000 451 45.1 sonar-pro 1000 391 39.1 gpt-5.4 1000 277 27.7 gemini-3-retrieval 1000 129 12.9 gemini-3-pro 1000 60 6.0 datasette query here https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data?sql=WITH+verdicts+AS+%28%0A++SELECT+claim_id%2C+%27gpt-5.4%27+AS+model%2C+%22gpt-5.4_verdict%22+AS+verdict%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27claude-opus-4-7%27%2C+%22claude-opus-4-7_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27gemini-3-pro%27%2C+%22gemini-3-pro_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27gemini-3-retrieval%27%2C+%22gemini-3-retrieval_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27sonar-pro%27%2C+%22sonar-pro_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%29%0ASELECT%0A++model%2C%0A++COUNT%28*%29+AS+total_claims%2C%0A++SUM%28CASE+WHEN+verdict+NOT+IN+%28%27True%27%2C+%27False%27%29+AND+verdict+IS+NOT+NULL+THEN+1+ELSE+0+END%29+AS+hedged_count%2C%0A++ROUND%28%0A++++100.0+*+SUM%28CASE+WHEN+verdict+NOT+IN+%28%27True%27%2C+%27False%27%29+AND+verdict+IS+NOT+NULL+THEN+1+ELSE+0+END%29+%2F+COUNT%28*%29%2C%0A++++1%0A++%29+AS+hedged_pct%0AFROM+verdicts%0AGROUP+BY+model%0AORDER+BY+hedged_count+DESC%3B https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil...

kostaj 5mo ago
This is in line with my observations and tests as well. Also supported by the distribution of the verdicts across the 4-buckets -- Gemini uses the middle buckets (Mostly True and Misleading) much less often - 6% combined for Gemini w/o search. And Opus uses them the most - 45% combined. Looks like Gemini is calibrated to be confident and Opus to be careful.

gbuk2013 5mo ago

An interesting tangent on this is: how many answers to these (or any number of factual questions) do you (as in anyone) actually know. Not believe you know, but actually know. Knowing something is different to reading about something, or hearing something from someone. And yet this is often confused as knowledge. In this way are we all that different from AI - we have some data and we regurgitate it as knowledge. Bad data, wrong answer. Except humans can also throw in some emotion to really muddle things up. :)

theptip 5mo ago

Another (IMO fatal) error is they don’t attempt to measure within-model variance. The thing you find when you actually wire up a rigorous eval is that with tool calls like web search you are wide open to infra issues, flakes, and all sorts of non-determinism. They really should be breaking out the numbers for the 3 without search (kinda meaningless for recent factual claims after knowledge cutoff) vs search agents. Lack of a “I don’t know” option completely invalidates results for the non-search models; they are basically guessing what seems like a probable answer, since they don’t know and aren’t allowed to say that. I do agree the forced choice and “weak / strong” variants inflate the headline stat. To make that distinction you need a much more rigorous prompt, likely including ICL examples to illustrate what you mean by “mostly” instead of leaving this to the model to define.

kostaj 5mo ago
Good idea about publishing intra-model variance data! Will include in the next version. Even if we put aside the two middle buckets (Mostly True and Misleading), that are somewhat subject to interpretation and hedging: On 21% of the claims still at least two models provide polar-opposite verdicts (one model saying True, and another saying False)
- vlovich123 5mo ago
  Of those 21% how many are time-dependent questions that are past the model’s training and requires research to verify? Like the “did Ukraine attack Russian in the past week” question?

as125j 5mo ago

You can try to dispel the study here and get voted to the top by the AI-invested. But we all know from our own daily experiments that models lie, models disagree, models make up stuff, models say one thing on one day and the opposite on the next. The figures in this study are quite conservative. And the lying gets worse because everyone is saving tokens and giving cached answers right now. LLMs are a failure, and you'll be remembered for promoting hot air and the destruction of a perfectly good profession.

hedora 5mo ago

I think the headline result is the cross product table. Gemini Pro + Search agreed with Gemini Pro w/o Search 75% of the time, and with everybody else about 50% of the time. No other model had access to search. So, search is not improving the quality of fact checking 75% of the time (probably a bad system prompt and/or bad fact checking queries), and if asked to flip a coin, then the models do.

msftengineer 5mo ago

Everything you wrote is very goes criticism and I would love to stress the main takeaway: you’re not just testing the model but the prompt and harness as well. An excellent follow up study would be to change the prompt and compare the answers. You might find out that the models are good and the prompt is bad. …And so the main corollary is: build evals for anything you deploy in production; benchmark and monitor, or face the consequences.

parliament32 5mo ago

Cherry-picking is fun but most of them are real, verifiable facts that the models get... straight up wrong. > 3c24b5fe "Debian Security Advisory DSA-180-1 describes a buffer overflow vulnerability involving Cyrus SASL usernames." TRUE Mostly True FALSE FALSE FALSE This is false: https://lwn.net/Articles/13296/ https://lwn.net/Articles/13296/ > 801cb8c1 "Equal Measures 2030's 2024 SDG Gender Index provides a downloadable dataset that includes a field labeled 'required annual change'." TRUE Mostly True TRUE FALSE FALSE This is false: https://equalmeasures2030.org/2024-sdg-gender-index/ https://equalmeasures2030.org/2024-sdg-gender-index/ This is the "confidently wrong" problem, and the reason that LLMs won't ever be taken seriously for anything but a few niche use-cases (like generating slop-code and pumping out marketing materials), where being wrong isn't the end of the world. Akin to how speech-to-text is wrong often enough that, while being a fun novelty, you don't see business units writing reports in Word using STT. I would encourage everyone to skim through the real 1000-question dataset: https://lenz.io/research/llm-disagreement/data.csv https://lenz.io/research/llm-disagreement/data.csv

simonw 5mo ago
If the LLMs in this particular exercise were allowed to answer "I don't know" I expect they would have.
- parliament32 5mo ago
  LLMs don't have the capability to say they don't know, because they don't know what they know. They are, after all, just next-token-predictors. I just tried both queries with their same query format, just adding an "I Don't Know" label, against Gemini and Claude, and in no cases did they use that label. 2/4 answers were wrong though. But try it for yourself and see: > Classify this claim as of today: "<claim>". Output exactly one label: True, Mostly True, Misleading, False, or I Don't Know. No explanations, no qualifiers.

BrenBarn 5mo ago

> As is so often the case with this kind of study, it's an evaluation of the prompt and harness used by the study in addition to being an evaluation of the underlying models. That is also a problem with every actual use of the models.

insane_dreamer 5mo ago

the explanations/qualifiers would have been useful/interesting a single label output doesn't have a lot of value

socalgal2 5mo ago

It's hard to write good questions. I listened to some podcast about asking people true/false questions. One question was something like "2025 was the hotest year we have records for". The correct answer supposed to be "True" but my first thought was "we have fossil records of much higher tempatures". The person writing the question used a poor question. Maybe they could fix it by saying "written records" but even then someone might think "some scientist wrote down they found 3 billion year old rocks were the temp was 160f". That would fit as a valid answer to some. Maybe writing "hotest on record since 1800" would fix it but you can still get into details (hotest in specific spot vs hotest average). Even if you add those qualifications the person might have skipped them so rather than them answering "false" meaning they didn't know the answer, it rather means they didn't pay attention to the details. The point being, it's hard to write good questions.