4 ms·

This is endlessly brought up as if the human operating the tool is an idiot. I agree that if the user is incompetent, cannot learn, and cannot learn to use a t

This is endlessly brought up as if the human operating the tool is an idiot. I agree that if the user is incompetent, cannot learn, and cannot learn to use a tool, then they're going to make a lot of mistakes from using GPTs. Yes, there are limitations to using GPTs. They are pre-trained, so of course they're not going to know about some easter egg in DDG. They are not an oracle. There is indeed skill to using them. They are not magic, so if that is the bar we expect them to hit, we will be disappointed. But neither are they useless, and it seems we constantly talk past one another because one side insists they're magic silicon gods, while the other says they're worthless because they are far short of that bar.

breve 1y ago
The ability to say "I don't know" is not a high bar. I would say it's a basic requirement of a system that is not magic.
- chrsw 1y ago
  > The ability to say "I don't know" is not a high bar. For you and I, it's not. But for these LLMs, maybe it's not that easy? They get their inputs, crunch their numbers, and come out with a confidence score. If they come up with an answer they're 99% confident in, by some stochastic stumbling through their weights, what are they supposed to do? I agree it's a problem that these systems are more likely to give poor, incorrect, or even obviously contradictory answers than say "I don't know". But for me, that's part of the risk of using these systems and that's why you need to be careful how you use them.
  - kuschku 1y ago
    but they're not. Ofyen the confidence value is much lower. I should have an option to see how confident it is. (maybe set the opacity of each token to its confidence?)
    - furyofantares 1y ago
      Logits aren't confidence about facts. You can turn on a display like this in the openai playground and you will see it doesn't do what you want.
  - johnnyanmac 1y ago
    >If they come up with an answer they're 99% confident in, by some stochastic stumbling through their weights, what are they supposed to do? As much as Fi, from The Legend of Zelda: Skyward Sword was mocked for this, this is the exact behavior a machine should do (not that Fi is a machine, but she operated as such). Give a confidence score the way we do in statistics, make sure to offer sources, and be ready to push back on more objective answers. accomplish those and I'd be way more comfortable using them as a tool. >hat's part of the risk of using these systems and that's why you need to be careful how you use them. Adn we know in 2025 how careful the general user is of consuming bias and propaganda, right?
    - otabdeveloper4 1y ago
      The confidence score is about the likelihood of this token appearing in this context. LLMs don't operate in facts or knowledge.
- throwawaylaptop 1y ago
  Based on your example, basically any answer would be "I don't know 100%". You could ask me as a human basically any question, and I'd have answers for most things I have experience with. But if you held a gun to head and said "are you sure???" I'd obviously answer "well damn, no I'm not THAT sure".
  - johnnyanmac 1y ago
    It'd at least be an honest one that recognizes that we shouldn't be trusting the tech wholesale yet. >But if you held a gun to head and said "are you sure???" I'd obviously answer "well damn, no I'm not THAT sure". okay, who's holding a gun to Sam Altman's head?
- cindyllrn 1y ago
  I see your point Some of the best exchanges that I participated in or witnessed involved people acknowledging their personal limits, including limits of conclusions formed a priori To further the discussion, hearing the phrase you mentioned would help the listener to independently assess a level of confidence or belief of the exchange But then again, honesty isn't on-brand for startups It's something that established companies say about themselves to differentiate from competitors or even past behavior of their own I mean, if someone prompted an llm weighted for honesty, who would pay for the following conversation? Prompt: can the plan as explained work? Response: I don't know about that. What I do know is on average, you're FUCKED.
- riwsky 1y ago
  Perhaps LLMs are magic?

breve 1y ago

The ability to say "I don't know" is not a high bar. I would say it's a basic requirement of a system that is not magic.

chrsw 1y ago
> The ability to say "I don't know" is not a high bar. For you and I, it's not. But for these LLMs, maybe it's not that easy? They get their inputs, crunch their numbers, and come out with a confidence score. If they come up with an answer they're 99% confident in, by some stochastic stumbling through their weights, what are they supposed to do? I agree it's a problem that these systems are more likely to give poor, incorrect, or even obviously contradictory answers than say "I don't know". But for me, that's part of the risk of using these systems and that's why you need to be careful how you use them.
- kuschku 1y ago
  but they're not. Ofyen the confidence value is much lower. I should have an option to see how confident it is. (maybe set the opacity of each token to its confidence?)
  - furyofantares 1y ago
    Logits aren't confidence about facts. You can turn on a display like this in the openai playground and you will see it doesn't do what you want.
- johnnyanmac 1y ago
  >If they come up with an answer they're 99% confident in, by some stochastic stumbling through their weights, what are they supposed to do? As much as Fi, from The Legend of Zelda: Skyward Sword was mocked for this, this is the exact behavior a machine should do (not that Fi is a machine, but she operated as such). Give a confidence score the way we do in statistics, make sure to offer sources, and be ready to push back on more objective answers. accomplish those and I'd be way more comfortable using them as a tool. >hat's part of the risk of using these systems and that's why you need to be careful how you use them. Adn we know in 2025 how careful the general user is of consuming bias and propaganda, right?
  - otabdeveloper4 1y ago
    The confidence score is about the likelihood of this token appearing in this context. LLMs don't operate in facts or knowledge.
throwawaylaptop 1y ago
Based on your example, basically any answer would be "I don't know 100%". You could ask me as a human basically any question, and I'd have answers for most things I have experience with. But if you held a gun to head and said "are you sure???" I'd obviously answer "well damn, no I'm not THAT sure".
- johnnyanmac 1y ago
  It'd at least be an honest one that recognizes that we shouldn't be trusting the tech wholesale yet. >But if you held a gun to head and said "are you sure???" I'd obviously answer "well damn, no I'm not THAT sure". okay, who's holding a gun to Sam Altman's head?
cindyllrn 1y ago
I see your point Some of the best exchanges that I participated in or witnessed involved people acknowledging their personal limits, including limits of conclusions formed a priori To further the discussion, hearing the phrase you mentioned would help the listener to independently assess a level of confidence or belief of the exchange But then again, honesty isn't on-brand for startups It's something that established companies say about themselves to differentiate from competitors or even past behavior of their own I mean, if someone prompted an llm weighted for honesty, who would pay for the following conversation? Prompt: can the plan as explained work? Response: I don't know about that. What I do know is on average, you're FUCKED.
riwsky 1y ago
Perhaps LLMs are magic?