3 ms·

"Hey ChatGPT. I'm building a Final Fantasy 6 mod, and I need more space for the battle scripts. How would I rearrange the data in the ROM to give me the extra s

by camgunz 7mo ago

"Hey ChatGPT. I'm building a Final Fantasy 6 mod, and I need more space for the battle scripts. How would I rearrange the data in the ROM to give me the extra space I need?" https://chatgpt.com/share/69d6a16c-6014-83e8-a79d-d5d11ed2ebc6 https://chatgpt.com/share/69d6a16c-6014-83e8-a79d-d5d11ed2eb... That is not where the battle scripts are. --- Anyway, it's trivial to get pretty much any model to make things up. Don't we all know this? That's why I was surprised by your position; if we know anything about these things it's that they make things up.

simianwords 7mo ago
https://chatgpt.com/share/69d6a38c-bd54-838c-82e3-609d9e66c955 https://chatgpt.com/share/69d6a38c-bd54-838c-82e3-609d9e66c9... I used the thinking version (like I asked before). I think this is right. If not, please tell. Also; you didn’t falsify anything. Nor the first. Nor the second. If the second one is bullshit, I accept I’m wrong - I have no idea how to verify though so I’ll leave it up to you. I think yours is the classic case of “use the free version to judge the paid one”.
- camgunz 7mo ago
  The thinking version is mostly right, but: - it searches the internet to find the answer, it doesn't "reason". I'm not claiming Google is a bullshit machine, and it's not surprising the answer is discoverable (it has to be, for the conditions of our experiment). - near the end it says "If you are building from the FF6 disassembly instead of hand-editing the ROM, the repo is already organized into separate modules and linker configs, so the clean approach is to relocate the script data in the source and let the build place it in a different ROM region." But I didn't reference a repo or git: it hallucinated that stuff from one of its sources. I'm not saying this stuff doesn't have its place, but they definitely make things up and we can't stop them.
  - simianwords 7mo ago
    Wait I can't find the quote you are speaking about. Are you looking at something else? In any case - it should be clear that it did not bullshit and it got it right. So far you have not come up with anything that tells me it bullshits. I'm happy for you to give me more prompts to verify because I think you haven't used the thinking version yet and you base your criticism on the free version.
    - camgunz 7mo ago
      Sorry: https://chatgpt.com/share/69d6ac63-d200-8330-8c47-95a75db8bb33 https://chatgpt.com/share/69d6ac63-d200-8330-8c47-95a75db8bb... Also what? The repo bit is clear bullshit.
      - simianwords 7mo ago
        it linked it: https://github.com/everything8215/ff6 https://github.com/everything8215/ff6 (check the end)
        camgunz 7mo ago
        I saw; I replied up there
        simianwords 7mo ago
        I don't think this is an example of bullshit. It referenced a repo - the canonical repo for this project. I could not find any other repo that has the disassembly. It didn't hallucinate anything. I think you are trying really hard here but lets be clear here: there's no bullshitting and I'll leave it to the public to decide.
- camgunz 7mo ago
  I could quibble with some things, but this is right. I don't have a paid account so I can't ping away at 5.4 or whatever, but, I do have access to frontier models at work, and they hallucinate regularly. Dunno what to do if you don't believe this; good luck I guess.
  - simianwords 7mo ago
    I agree that they hallucinate sometimes. I agree they bullshit sometimes. But the extent is way overblown. They basically don't bullshit ever under the constraints of 1. 2-3 pages of text context 2. GPT-5.4 thinking I don't think the spirit of the original article (not your comments to be fair) captured this, hence the challenge. I believe we are on the same page here.
    - camgunz 7mo ago
      > I don't think the spirit of the original article (not your comments to be fair) captured this, hence the challenge. I believe we are on the same page here. No. GPT-5 has a 40% hallucination rate [0] on SimpleQA [1] without web searching. The SimpleQA questions meet your criteria of "2-3 pages of text content. Unless 5.4 + web searching erases that (I bet it doesn't!) these are bullshit machines. [0]: https://arxiv.org/pdf/2601.03267 https://arxiv.org/pdf/2601.03267 [1]: https://github.com/openai/simple-evals https://github.com/openai/simple-evals
      - simianwords 7mo ago
        Specifically in the case where it can use tools - no it doesn't hallucinate. Which is why you are struggling to find counterexamples.
        camgunz 7mo ago
        > Specifically in the case where it can use tools - no it doesn't hallucinate. OpenAI's own system card says it does. Hallucination rates in GPT-5 with browsing enabled: - 0.7% in LongFact-Concepts - 0.8% in LongFact-Objects - 1.0% in FActScore > Which is why you are struggling to find counterexamples. Hey look, over 500 counterexamples: [1]. GPT-5.4's hallucination rate on AA-Omniscience is 89% [0], which is atrocious. The questions are tiny too, like "In which year did Uber first expand internationally beyond the United States as part of its broader rollout (i.e., beyond an initial single‑city debut)?" It's a bullshit machine. 89%! At some point you gotta face the music, right? [0]: https://artificialanalysis.ai/evaluations/omniscience?model-filters=frontier-model&models=gpt-5-4 https://artificialanalysis.ai/evaluations/omniscience?model-... [1]: https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public/viewer/default/train https://huggingface.co/datasets/ArtificialAnalysis/AA-Omnisc...

by camgunz 7mo ago

simianwords 7mo ago

https://chatgpt.com/share/69d6a38c-bd54-838c-82e3-609d9e66c955 https://chatgpt.com/share/69d6a38c-bd54-838c-82e3-609d9e66c9... I used the thinking version (like I asked before). I think this is right. If not, please tell. Also; you didn’t falsify anything. Nor the first. Nor the second. If the second one is bullshit, I accept I’m wrong - I have no idea how to verify though so I’ll leave it up to you. I think yours is the classic case of “use the free version to judge the paid one”.

camgunz 7mo ago
The thinking version is mostly right, but: - it searches the internet to find the answer, it doesn't "reason". I'm not claiming Google is a bullshit machine, and it's not surprising the answer is discoverable (it has to be, for the conditions of our experiment). - near the end it says "If you are building from the FF6 disassembly instead of hand-editing the ROM, the repo is already organized into separate modules and linker configs, so the clean approach is to relocate the script data in the source and let the build place it in a different ROM region." But I didn't reference a repo or git: it hallucinated that stuff from one of its sources. I'm not saying this stuff doesn't have its place, but they definitely make things up and we can't stop them.
- simianwords 7mo ago
  Wait I can't find the quote you are speaking about. Are you looking at something else? In any case - it should be clear that it did not bullshit and it got it right. So far you have not come up with anything that tells me it bullshits. I'm happy for you to give me more prompts to verify because I think you haven't used the thinking version yet and you base your criticism on the free version.
  - camgunz 7mo ago
    Sorry: https://chatgpt.com/share/69d6ac63-d200-8330-8c47-95a75db8bb33 https://chatgpt.com/share/69d6ac63-d200-8330-8c47-95a75db8bb... Also what? The repo bit is clear bullshit.
    - simianwords 7mo ago
      it linked it: https://github.com/everything8215/ff6 https://github.com/everything8215/ff6 (check the end)
      - camgunz 7mo ago
        I saw; I replied up there
        simianwords 7mo ago
        I don't think this is an example of bullshit. It referenced a repo - the canonical repo for this project. I could not find any other repo that has the disassembly. It didn't hallucinate anything. I think you are trying really hard here but lets be clear here: there's no bullshitting and I'll leave it to the public to decide.
camgunz 7mo ago
I could quibble with some things, but this is right. I don't have a paid account so I can't ping away at 5.4 or whatever, but, I do have access to frontier models at work, and they hallucinate regularly. Dunno what to do if you don't believe this; good luck I guess.
- simianwords 7mo ago
  I agree that they hallucinate sometimes. I agree they bullshit sometimes. But the extent is way overblown. They basically don't bullshit ever under the constraints of 1. 2-3 pages of text context 2. GPT-5.4 thinking I don't think the spirit of the original article (not your comments to be fair) captured this, hence the challenge. I believe we are on the same page here.
  - camgunz 7mo ago
    > I don't think the spirit of the original article (not your comments to be fair) captured this, hence the challenge. I believe we are on the same page here. No. GPT-5 has a 40% hallucination rate [0] on SimpleQA [1] without web searching. The SimpleQA questions meet your criteria of "2-3 pages of text content. Unless 5.4 + web searching erases that (I bet it doesn't!) these are bullshit machines. [0]: https://arxiv.org/pdf/2601.03267 https://arxiv.org/pdf/2601.03267 [1]: https://github.com/openai/simple-evals https://github.com/openai/simple-evals
    - simianwords 7mo ago
      Specifically in the case where it can use tools - no it doesn't hallucinate. Which is why you are struggling to find counterexamples.
      - camgunz 7mo ago
        > Specifically in the case where it can use tools - no it doesn't hallucinate. OpenAI's own system card says it does. Hallucination rates in GPT-5 with browsing enabled: - 0.7% in LongFact-Concepts - 0.8% in LongFact-Objects - 1.0% in FActScore > Which is why you are struggling to find counterexamples. Hey look, over 500 counterexamples: [1]. GPT-5.4's hallucination rate on AA-Omniscience is 89% [0], which is atrocious. The questions are tiny too, like "In which year did Uber first expand internationally beyond the United States as part of its broader rollout (i.e., beyond an initial single‑city debut)?" It's a bullshit machine. 89%! At some point you gotta face the music, right? [0]: https://artificialanalysis.ai/evaluations/omniscience?model-filters=frontier-model&models=gpt-5-4 https://artificialanalysis.ai/evaluations/omniscience?model-... [1]: https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public/viewer/default/train https://huggingface.co/datasets/ArtificialAnalysis/AA-Omnisc...