5 ms·

At this point the single biggest improvement that could be made to GPTs is making them able to say "I don't know" when they honestly don't. Just today I was pl

At this point the single biggest improvement that could be made to GPTs is making them able to say "I don't know" when they honestly don't. Just today I was playing around with modding Cyberpunk 2077 and was looking for a way to programmatically spawn NPCs in redscript. It was hard to figure out, but I managed. ChatGPT 5 just hallucinated some APIs even after doing "research" and repeatedly being called out. After 30 minutes of ChatGPT wasting my time I accepted that I'm on my own. It could've been 1 minute.

mupuff1234 1y ago
Yeah I'm surprised that there's not at least some sort of conviction metric being outputted along the LLM response. I mean it's all probability right? Must be a way to give it some score.
- bravesoul2 1y ago
  Not sure. In RLHF you are adjusting the weights away from wrong answers in general. So this is being done. I think the closest you can get without more research is another model checking the answer and looking for BS. This will cripple speed but if it can be more agentic and async it may not matter. I think people need to choose between chat interface and better answers.
yosito 1y ago
Don't make the mistake of thinking that "knowing" has anything to do with the output of ChatGPT. It gives you the statistically most likely output based on its training data. It's not checking some sort of internal knowledge system, it's literally just outputting statistical linguistic patterns. This technology can be trained to emphasize certain ideas (like propaganda) but it can not be used directly to access knowledge.
- chmod775 1y ago
  > It's not checking some sort of internal knowledge system In my case it was consuming online sources, then repeating "information" not actually contained therein. This, at least, is absolutely preventable even without any metacognition to speak of.
efnx 1y ago
I totally agree. That would be great. I think the problem with that is LLMs don’t know what they don’t know. It’s arguable they even “know” anything!
bravesoul2 1y ago
Like the XKCD reference but bigger: Give me a 100bn research team and 25 years.
crindy 1y ago
They do talk about working on this, and making improvements. From https://openai.com/index/introducing-gpt-5/ https://openai.com/index/introducing-gpt-5/ > More honest responses > Alongside improved factuality, GPT‑5 (with thinking) more honestly communicates its actions and capabilities to the user—especially for tasks which are impossible, underspecified, or missing key tools. In order to achieve a high reward during training, reasoning models may learn to lie about successfully completing a task or be overly confident about an uncertain answer. For example, to test this, we removed all the images from the prompts of the multimodal benchmark CharXiv, and found that OpenAI o3 still gave confident answers about non-existent images 86.7% of the time, compared to just 9% for GPT‑5. > When reasoning, GPT‑5 more accurately recognizes when tasks can’t be completed and communicates its limits clearly. We evaluated deception rates on settings involving impossible coding tasks and missing multimodal assets, and found that GPT‑5 (with thinking) is less deceptive than o3 across the board. On a large set of conversations representative of real production ChatGPT traffic, we’ve reduced rates of deception from 4.8% for o3 to 2.1% of GPT‑5 reasoning responses. While this represents a meaningful improvement for users, more work remains to be done, and we’re continuing research into improving the factuality and honesty of our models. Further details can be found in the system card.
arolihas 1y ago
It doesn't "know" anything. Everything that comes out is a hallucination contingent on the prompt.
- brokencode 1y ago
  You could say the same about humans. Have you ever misremembered something that you thought you knew? Sure, typically we don’t invent totally made up names, but we certainly do make mistakes. Our memory can be quite hazy and unreliable as well.
  - arolihas 1y ago
    Do you genuinely believe that humans just hallucinate everything? When you or I say my favorite ice cream flavor is vanilla, is that just a hallucination? If ChatGPT were to say their favorite ice cream flavor is vanilla, are you taking it with equal weight? Come on.
    - brokencode 1y ago
      I genuinely believe that human brains are made of neurons and that our memories arise from how those neurons connect. I believe this is fundamentally lossy and probabilistic. Obviously human brains are still much more sophisticated than the artificial neural networks that we can make with current technology. But I believe there’s a lot more in common than some people would like to admit.
      - arolihas 1y ago
        Ok, that is memory. I am talking about hallucination vs human or even animal intent in an embodied meaningful experience.
        brokencode 1y ago
        All you’re doing is calling the same thing hallucination when an LLM does it and memory when a human does it. You have provided no basis that the two are actually different. Humans are better at noticing when their recollections are incorrect. But LLMs are quickly improving.
abrookewood 1y ago
Yep, that's a great point. They often feel like a co-worker who speaks with such complete authority on a subject that you don't even consider alternatives, until you realise they are lying. Extremely frustrating.
- vdfs 1y ago
  That is extremely hard, It require the model to have "knowledge" so it can decide if it know the answer or not which is not how current llm/gpt works
PessimalDecimal 1y ago
> At this point the single biggest improvement that could be made to GPTs is making them able to say "I don't know" when they honestly don't. You're not alone in thinking this. And I'm sure this has been considered within the frontier AI labs and surely has been tried. The fact that it's so uncommon must mean something about what these models are capable of, right?
- hodgehog11 1y ago
  Yes, there are people working on this, but not as many as one would like. GPTs have uncertainty baked into them, but the problem is that it's for the next-token prediction task and not for the response as a whole.
pamelafox 1y ago
I just ran evaluations of gpt-5 for our RAG scenario and was pleasantly surprised at how often it admitted “ I don’t know” - more than any model I’ve eval’d before. Our prompt does tell it to say it doesnt know if context is missing, so that likely helped, but this is the first model to really adhere to that.
booleandilemma 1y ago
They're like some of the overconfident people I've worked with who are too insecure to say they don't know to our boss.
freediver 1y ago
"I do not know" is rarely in the training data as a follow up to anything.

mupuff1234 1y ago

Yeah I'm surprised that there's not at least some sort of conviction metric being outputted along the LLM response. I mean it's all probability right? Must be a way to give it some score.

bravesoul2 1y ago
Not sure. In RLHF you are adjusting the weights away from wrong answers in general. So this is being done. I think the closest you can get without more research is another model checking the answer and looking for BS. This will cripple speed but if it can be more agentic and async it may not matter. I think people need to choose between chat interface and better answers.

yosito 1y ago

Don't make the mistake of thinking that "knowing" has anything to do with the output of ChatGPT. It gives you the statistically most likely output based on its training data. It's not checking some sort of internal knowledge system, it's literally just outputting statistical linguistic patterns. This technology can be trained to emphasize certain ideas (like propaganda) but it can not be used directly to access knowledge.

chmod775 1y ago
> It's not checking some sort of internal knowledge system In my case it was consuming online sources, then repeating "information" not actually contained therein. This, at least, is absolutely preventable even without any metacognition to speak of.

efnx 1y ago

I totally agree. That would be great. I think the problem with that is LLMs don’t know what they don’t know. It’s arguable they even “know” anything!

bravesoul2 1y ago

Like the XKCD reference but bigger: Give me a 100bn research team and 25 years.

crindy 1y ago

They do talk about working on this, and making improvements. From https://openai.com/index/introducing-gpt-5/ https://openai.com/index/introducing-gpt-5/ > More honest responses > Alongside improved factuality, GPT‑5 (with thinking) more honestly communicates its actions and capabilities to the user—especially for tasks which are impossible, underspecified, or missing key tools. In order to achieve a high reward during training, reasoning models may learn to lie about successfully completing a task or be overly confident about an uncertain answer. For example, to test this, we removed all the images from the prompts of the multimodal benchmark CharXiv, and found that OpenAI o3 still gave confident answers about non-existent images 86.7% of the time, compared to just 9% for GPT‑5. > When reasoning, GPT‑5 more accurately recognizes when tasks can’t be completed and communicates its limits clearly. We evaluated deception rates on settings involving impossible coding tasks and missing multimodal assets, and found that GPT‑5 (with thinking) is less deceptive than o3 across the board. On a large set of conversations representative of real production ChatGPT traffic, we’ve reduced rates of deception from 4.8% for o3 to 2.1% of GPT‑5 reasoning responses. While this represents a meaningful improvement for users, more work remains to be done, and we’re continuing research into improving the factuality and honesty of our models. Further details can be found in the system card.

arolihas 1y ago

It doesn't "know" anything. Everything that comes out is a hallucination contingent on the prompt.

brokencode 1y ago
You could say the same about humans. Have you ever misremembered something that you thought you knew? Sure, typically we don’t invent totally made up names, but we certainly do make mistakes. Our memory can be quite hazy and unreliable as well.
- arolihas 1y ago
  Do you genuinely believe that humans just hallucinate everything? When you or I say my favorite ice cream flavor is vanilla, is that just a hallucination? If ChatGPT were to say their favorite ice cream flavor is vanilla, are you taking it with equal weight? Come on.
  - brokencode 1y ago
    I genuinely believe that human brains are made of neurons and that our memories arise from how those neurons connect. I believe this is fundamentally lossy and probabilistic. Obviously human brains are still much more sophisticated than the artificial neural networks that we can make with current technology. But I believe there’s a lot more in common than some people would like to admit.
    - arolihas 1y ago
      Ok, that is memory. I am talking about hallucination vs human or even animal intent in an embodied meaningful experience.
      - brokencode 1y ago
        All you’re doing is calling the same thing hallucination when an LLM does it and memory when a human does it. You have provided no basis that the two are actually different. Humans are better at noticing when their recollections are incorrect. But LLMs are quickly improving.

abrookewood 1y ago

Yep, that's a great point. They often feel like a co-worker who speaks with such complete authority on a subject that you don't even consider alternatives, until you realise they are lying. Extremely frustrating.

vdfs 1y ago
That is extremely hard, It require the model to have "knowledge" so it can decide if it know the answer or not which is not how current llm/gpt works

PessimalDecimal 1y ago

> At this point the single biggest improvement that could be made to GPTs is making them able to say "I don't know" when they honestly don't. You're not alone in thinking this. And I'm sure this has been considered within the frontier AI labs and surely has been tried. The fact that it's so uncommon must mean something about what these models are capable of, right?

hodgehog11 1y ago
Yes, there are people working on this, but not as many as one would like. GPTs have uncertainty baked into them, but the problem is that it's for the next-token prediction task and not for the response as a whole.

pamelafox 1y ago

I just ran evaluations of gpt-5 for our RAG scenario and was pleasantly surprised at how often it admitted “ I don’t know” - more than any model I’ve eval’d before. Our prompt does tell it to say it doesnt know if context is missing, so that likely helped, but this is the first model to really adhere to that.

booleandilemma 1y ago

They're like some of the overconfident people I've worked with who are too insecure to say they don't know to our boss.

freediver 1y ago

"I do not know" is rarely in the training data as a follow up to anything.