3 ms·

> I don't think the spirit of the original article (not your comments to be fair) captured this, hence the challenge. I believe we are on the same page here. N

by camgunz 7mo ago

> I don't think the spirit of the original article (not your comments to be fair) captured this, hence the challenge. I believe we are on the same page here. No. GPT-5 has a 40% hallucination rate [0] on SimpleQA [1] without web searching. The SimpleQA questions meet your criteria of "2-3 pages of text content. Unless 5.4 + web searching erases that (I bet it doesn't!) these are bullshit machines. [0]: https://arxiv.org/pdf/2601.03267 https://arxiv.org/pdf/2601.03267 [1]: https://github.com/openai/simple-evals https://github.com/openai/simple-evals

simianwords 7mo ago
Specifically in the case where it can use tools - no it doesn't hallucinate. Which is why you are struggling to find counterexamples.
- camgunz 7mo ago
  > Specifically in the case where it can use tools - no it doesn't hallucinate. OpenAI's own system card says it does. Hallucination rates in GPT-5 with browsing enabled: - 0.7% in LongFact-Concepts - 0.8% in LongFact-Objects - 1.0% in FActScore > Which is why you are struggling to find counterexamples. Hey look, over 500 counterexamples: [1]. GPT-5.4's hallucination rate on AA-Omniscience is 89% [0], which is atrocious. The questions are tiny too, like "In which year did Uber first expand internationally beyond the United States as part of its broader rollout (i.e., beyond an initial single‑city debut)?" It's a bullshit machine. 89%! At some point you gotta face the music, right? [0]: https://artificialanalysis.ai/evaluations/omniscience?model-filters=frontier-model&models=gpt-5-4 https://artificialanalysis.ai/evaluations/omniscience?model-... [1]: https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public/viewer/default/train https://huggingface.co/datasets/ArtificialAnalysis/AA-Omnisc...
  - simianwords 7mo ago
    You had to go all the way and find it in the benchmark results that specifically stress test this. You could not come up with a single one yourself. And you also linked an example where it was not allowed to use tools when I specifically said that it should be able to use tools. I'm not sure why are you present this as though it is a big gotcha. I think my main point pretty much stands.
    - camgunz 7mo ago
      I found over 500 examples that fit your criteria. Embarrassing you were arguing in bad faith this whole time.
      - simianwords 7mo ago
        They all use the tool search, no? Please correct me if I'm wrong. My criteria was using ChatGPT which explicitly allows it. https://arxiv.org/html/2511.13029v1 https://arxiv.org/html/2511.13029v1 if you don't believe me. BTW this was your original point >Anyway, it's trivial to get pretty much any model to make things up. Don't we all know this? That's why I was surprised by your position; if we know anything about these things it's that they make things up. And look at how much effort you have had to do 1. use the wrong model for the horns example 2. the game one also didn't work 3. now you are searching for examples in literal benchmarks and you are still not able to find any How is this trivial in any interpretation of the word? I think it would be perfectly reasonable to agree that it is not at all trivial to find counter examples for my challenge.
        camgunz 7mo ago
        I've got about 20 minutes in this; mostly I've been reading wallstreetbets at the Shake Shack bar in the Boston airport. I'm happy to post this over and over again until you engage w/ it: > I found over 500 examples that fit your criteria.
        simianwords 7mo ago
        They don't use tools. Like the 4th time you ignored this on purpose. That was not part of the challenge.
        camgunz 7mo ago
        GPT-5.4 gets 82.7% on Browsecomp (a benchmark specifically testing tool use), which is a hallucination rate of 17.3%, on questions like "Give me the title of the scientific paper published in the EMNLP conference between 2018-2023 where the first author did their undergrad at Dartmouth College and the fourth author did their undergrad at University of Pennsylvania." Since the goalposts have been moved to include effort, I'm compelled to say I found this while waiting in line at Starbucks, 5 mins tops. Probably GPT-5.4 could have found this too, though it lies > 1/6 the time, so one could be forgiven for not wanting to risk it. https://llm-stats.com/benchmarks/browsecomp https://llm-stats.com/benchmarks/browsecomp https://openai.com/index/browsecomp/ https://openai.com/index/browsecomp/

by camgunz 7mo ago

simianwords 7mo ago

Specifically in the case where it can use tools - no it doesn't hallucinate. Which is why you are struggling to find counterexamples.

camgunz 7mo ago
> Specifically in the case where it can use tools - no it doesn't hallucinate. OpenAI's own system card says it does. Hallucination rates in GPT-5 with browsing enabled: - 0.7% in LongFact-Concepts - 0.8% in LongFact-Objects - 1.0% in FActScore > Which is why you are struggling to find counterexamples. Hey look, over 500 counterexamples: [1]. GPT-5.4's hallucination rate on AA-Omniscience is 89% [0], which is atrocious. The questions are tiny too, like "In which year did Uber first expand internationally beyond the United States as part of its broader rollout (i.e., beyond an initial single‑city debut)?" It's a bullshit machine. 89%! At some point you gotta face the music, right? [0]: https://artificialanalysis.ai/evaluations/omniscience?model-filters=frontier-model&models=gpt-5-4 https://artificialanalysis.ai/evaluations/omniscience?model-... [1]: https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public/viewer/default/train https://huggingface.co/datasets/ArtificialAnalysis/AA-Omnisc...
- simianwords 7mo ago
  You had to go all the way and find it in the benchmark results that specifically stress test this. You could not come up with a single one yourself. And you also linked an example where it was not allowed to use tools when I specifically said that it should be able to use tools. I'm not sure why are you present this as though it is a big gotcha. I think my main point pretty much stands.
  - camgunz 7mo ago
    I found over 500 examples that fit your criteria. Embarrassing you were arguing in bad faith this whole time.
    - simianwords 7mo ago
      They all use the tool search, no? Please correct me if I'm wrong. My criteria was using ChatGPT which explicitly allows it. https://arxiv.org/html/2511.13029v1 https://arxiv.org/html/2511.13029v1 if you don't believe me. BTW this was your original point >Anyway, it's trivial to get pretty much any model to make things up. Don't we all know this? That's why I was surprised by your position; if we know anything about these things it's that they make things up. And look at how much effort you have had to do 1. use the wrong model for the horns example 2. the game one also didn't work 3. now you are searching for examples in literal benchmarks and you are still not able to find any How is this trivial in any interpretation of the word? I think it would be perfectly reasonable to agree that it is not at all trivial to find counter examples for my challenge.
      - camgunz 7mo ago
        I've got about 20 minutes in this; mostly I've been reading wallstreetbets at the Shake Shack bar in the Boston airport. I'm happy to post this over and over again until you engage w/ it: > I found over 500 examples that fit your criteria.
        simianwords 7mo ago
        They don't use tools. Like the 4th time you ignored this on purpose. That was not part of the challenge.
        camgunz 7mo ago
        GPT-5.4 gets 82.7% on Browsecomp (a benchmark specifically testing tool use), which is a hallucination rate of 17.3%, on questions like "Give me the title of the scientific paper published in the EMNLP conference between 2018-2023 where the first author did their undergrad at Dartmouth College and the fourth author did their undergrad at University of Pennsylvania." Since the goalposts have been moved to include effort, I'm compelled to say I found this while waiting in line at Starbucks, 5 mins tops. Probably GPT-5.4 could have found this too, though it lies > 1/6 the time, so one could be forgiven for not wanting to risk it. https://llm-stats.com/benchmarks/browsecomp https://llm-stats.com/benchmarks/browsecomp https://openai.com/index/browsecomp/ https://openai.com/index/browsecomp/