4 ms·

Disagree. You have to try really hard and go very niche and deep for it to get some fact wrong. In fact I'll ask you to provide examples: use GPT 5 with thinkin

Disagree. You have to try really hard and go very niche and deep for it to get some fact wrong. In fact I'll ask you to provide examples: use GPT 5 with thinking and search disabled and get it to give you inaccurate facts for non niche, non deep topics. Non niche meaning: something that is taught at undergraduate level and relatively popular. Non deep meaning you aren't going so deep as to confuse even humans. Like solving an extremely hard integral. Edit: probably a bad idea because this sort of "challenge" works only statistically not anecdotally. Still interesting to find out.

JustExAWS 1y ago
I literally just had ChatGPT create a Python program and it used .ends_with instead of .endswith. This was with ChatGPT 5. I mean it got a generic built in function of one of the most popular languages in the world wrong.
- simianwords 1y ago
  "but using LLMs for answering factual questions" this was about fact checking. Of course I know LLM's are going to hallucinate in coding sometimes.
  - JustExAWS 1y ago
    So it isn’t a “fact” that the built in Python function that tests whether a string ends with a substring is “endswith”? See https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect If you know that a source isn’t to be believed in an area you know about, why would you trust that source in an area you don’t know about? Another funny anecdote, ChatGPT just got the Gell-Man effect wrong. https://chatgpt.com/share/68a0b7af-5e40-8010-b1e3-ee9ff3c8cb6b https://chatgpt.com/share/68a0b7af-5e40-8010-b1e3-ee9ff3c8cb...
    - simianwords 1y ago
      It got it right with thinking which was the challenge I posed. https://chatgpt.com/share/68a0b897-f8dc-800b-8799-9be2a8ad5433 https://chatgpt.com/share/68a0b897-f8dc-800b-8799-9be2a8ad54...
      - OnlineGladiator 1y ago
        The point you're missing is it's not always right. Cherry-picking examples doesn't really bolster your point. Obviously it works for you (or at least you think it does), but I can confidently say it's fucking god-awful for me.
        simianwords 1y ago
        Am I really the one cherry picking? Please read the thread.
        OnlineGladiator 1y ago
        Yes. If someone gives an example of it not working, and you reply "but that example worked for me" then you're cherry picking when it works. Just because it worked for you does not mean it works for other people. If I ask ChatGPT a question and it gives me a wrong answer, ChatGPT is the fucking problem.
        simianwords 1y ago
        The poster didn't use "thinking" model. That was my original challenge!! Why don't you try the original prompt using thinking model and see if I'm cherry picking?
        OnlineGladiator 1y ago
        Every time I use ChatGPT I become incredibly frustrated with how fucking awful it is. I've used it more than enough, time and time again (just try the new model, bro!), to know that I fucking hate it. If it works for you, cool. I think it's dogshit.
malfist 1y ago
Maybe you should fact check your AI outputs more if you think it only hallucinates in niche topics
- simianwords 1y ago
  The accuracy is high enough that I don't have to fact check too often.
  - collingreen 1y ago
    Without some exploratory fact checking how do you estimate how high the accuracy is and how often you should be fact checking to maintain a good understanding?
    - simianwords 1y ago
      I did initial tests so that I don't have to do it anymore.
      - malfist 1y ago
        If there's one thing that's constant it's that these systems change.
      - jibal 1y ago
        Everyone else has done tests that indicate that you do.
        glenstein 1y ago
        And this is why you can't use personal anecdotes to settle questions of software performance. Comment sections are never good at being accountable for how vibes-driven they are when selecting which anecdotes to prefer.
  - platevoltage 1y ago
    I totally get that you meant this in a nuanced way, but at face value it sort of reads like... Joe Rogan has high enough accuracy that I don't have to fact check too often. Newsmax has high enough accuracy that I don't have to fact check too often, etc. If you accept the output as accurate, why would fact checking even cross your mind?

JustExAWS 1y ago

I literally just had ChatGPT create a Python program and it used .ends_with instead of .endswith. This was with ChatGPT 5. I mean it got a generic built in function of one of the most popular languages in the world wrong.

simianwords 1y ago
"but using LLMs for answering factual questions" this was about fact checking. Of course I know LLM's are going to hallucinate in coding sometimes.
- JustExAWS 1y ago
  So it isn’t a “fact” that the built in Python function that tests whether a string ends with a substring is “endswith”? See https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect If you know that a source isn’t to be believed in an area you know about, why would you trust that source in an area you don’t know about? Another funny anecdote, ChatGPT just got the Gell-Man effect wrong. https://chatgpt.com/share/68a0b7af-5e40-8010-b1e3-ee9ff3c8cb6b https://chatgpt.com/share/68a0b7af-5e40-8010-b1e3-ee9ff3c8cb...
  - simianwords 1y ago
    It got it right with thinking which was the challenge I posed. https://chatgpt.com/share/68a0b897-f8dc-800b-8799-9be2a8ad5433 https://chatgpt.com/share/68a0b897-f8dc-800b-8799-9be2a8ad54...
    - OnlineGladiator 1y ago
      The point you're missing is it's not always right. Cherry-picking examples doesn't really bolster your point. Obviously it works for you (or at least you think it does), but I can confidently say it's fucking god-awful for me.
      - simianwords 1y ago
        Am I really the one cherry picking? Please read the thread.
        OnlineGladiator 1y ago
        Yes. If someone gives an example of it not working, and you reply "but that example worked for me" then you're cherry picking when it works. Just because it worked for you does not mean it works for other people. If I ask ChatGPT a question and it gives me a wrong answer, ChatGPT is the fucking problem.
        simianwords 1y ago
        The poster didn't use "thinking" model. That was my original challenge!! Why don't you try the original prompt using thinking model and see if I'm cherry picking?
        OnlineGladiator 1y ago
        Every time I use ChatGPT I become incredibly frustrated with how fucking awful it is. I've used it more than enough, time and time again (just try the new model, bro!), to know that I fucking hate it. If it works for you, cool. I think it's dogshit.

malfist 1y ago

Maybe you should fact check your AI outputs more if you think it only hallucinates in niche topics

simianwords 1y ago
The accuracy is high enough that I don't have to fact check too often.
- collingreen 1y ago
  Without some exploratory fact checking how do you estimate how high the accuracy is and how often you should be fact checking to maintain a good understanding?
  - simianwords 1y ago
    I did initial tests so that I don't have to do it anymore.
    - malfist 1y ago
      If there's one thing that's constant it's that these systems change.
    - jibal 1y ago
      Everyone else has done tests that indicate that you do.
      - glenstein 1y ago
        And this is why you can't use personal anecdotes to settle questions of software performance. Comment sections are never good at being accountable for how vibes-driven they are when selecting which anecdotes to prefer.
- platevoltage 1y ago
  I totally get that you meant this in a nuanced way, but at face value it sort of reads like... Joe Rogan has high enough accuracy that I don't have to fact check too often. Newsmax has high enough accuracy that I don't have to fact check too often, etc. If you accept the output as accurate, why would fact checking even cross your mind?