3 ms·

Hallucinating is fine but overconfidence is the problem. But I heard it's not an easy problem to solve.

by admiralrohan 2y ago

jacksnipe 2y ago
Unfortunately, in the system most of us work in today, I think overconfidence is an intelligent behavior.
- onemoresoop 2y ago
  I find it extremely dumb to see overconfident people that really have nothing special about them or are even incompetent. These people are not contributing positively to the system, quite on the contrary.
  - xwolfi 2y ago
    But we don't work for the system, fundamentally, we work for ourselves, and the system incentivizes us to work for it by aligning our constraints: if you work that direction, you'll get that reward. Overconfident people ofc do not contribute positively to the system, but they skew the system reward's calculation towards them: I swear I've done that work in that direction, where's my reward ? In a sense, they are extremely successful: they managed to do very low effort, get very high reward, help themselves like all of us but at a much better profit margin, by sacrificing a system that, let's be honest, none of us care about really. Your problem maybe, is that you swallowed the little BS the system fed you while incentivizing you: that the system matters more than yourself, at least at a greater extent than healthy ? And you see the same thing with AI: these things convince people so deeply of their intelligence that it blew to such proportion that NVidia is now worth trillions. I had a colleague mumbling yesterday that his wife now speaks more with ChatGPT than him. Overconfidence is a positive attribute... for oneself.
    - onemoresoop 2y ago
      Not talking about THE system or any system in particular same way gaming the system doesn’t refer to any system but just cheating in general. And if you like overconfident people good for you, I can’t stand them because they’re hollow with no basis in reality, flawed like everyone, just pumping out their egos with hot air. And your reasoning that overconfidence is a positive attribute doesn’t make much sense to me but we’re entitled to our own opinions.
    - butlike 2y ago
      Overconfident people are conquerors. Conquerors do not contribute positively to a harmonious system, true, but I'm not so sure we can glean the system is supposed to be harmonious. If one contributes "positively" to the system, everyone's value increases and the solution becomes more homogenized. Once the system is homogenized enough, it becomes vulnerable to adversity from an outside force. If the system is not harmonious/non-homoginized, the attacker would be drawn to the most powerful point in the system. Overconfident people aren't evil, they're simply stressing the system to make sure it can handle adversity from an outside force. They're saying: "listen, I'm going to take what you have, and you should be so happy that's all I'm taking." So I think overconfidence is a positive attribute for the system as well as for the overconfident individual. It's not a positive attribute for the local parties getting run over by the overconfident individual.
    - jacksnipe 2y ago
      Yeah this is what I meant, both in the behavior being intelligent and it being unfortunate that this is the case. It’d be nice if the most self-maximizing behavior were also the best behavior for the global system, but it doesn’t seem that it is.
  - tomComb 2y ago
    But being like that can get you elected.
    - Freedom2 2y ago
      In what country?
      - tbossanova 2y ago
        Quite a few :(
      - chromehearts 2y ago
        All of them
    - sim7c00 2y ago
      and getting yourself elected while being underqualified is intelligent? i think its not. its stupid and damaging behavior based in selfish desires. about as far from intelligent you can get.
      - rcxdude 2y ago
        Intelligence is seperate from goals: if you're only interested in gaining power and wealth for yourself, then concern about the rest of the system is only incidental to what you can get for yourself.
    - throw4847285 2y ago
      That's a fallacy. There are certainly some unqualified elected leaders, but humans living in democratic societies have yet to shake the mental framework we've constructed from centuries without self-rule. We invest way more authority into a single executive than they actually have, and blame them for everything that goes wrong despite the fact that modern democracies are hugely complex systems in which authority is distributed across numerous people. When the government fails to meet people's needs, they lack the capacity to point at a particular Senator or a small party in a ruling coalition and blame them. It's always the executive. Of course, the result is that people get fed up and decide that the problem has been not that democratic societies are hard to govern by design (they have to reflect the disparate desires of countless people) but that the executive was too weak. They get behind whatever candidate is charismatic enough to convince them that they will govern the way the people already thought the previous executives were governing, just badly. The result is an incompetent tyrant.
  - butlike 2y ago
    Overconfidence allows you to act, increasing survivability. Thinking is a "weak" trait.
Terr_ 2y ago
> overconfidence is the problem. The problem is a bit deeper than that, because what we perceive as "confidence" is itself also an illusion. The (real) algorithm takes documents and makes them longer, and some humans configured a document that looks like a conversation between "User" and "AssistantBot", and they also wrote some code to act-out things that look like dialogue for one of the characters. The (real) trait of confidence involves next-token statistics. In contrast, the character named AssistantBot is "overconfident" in exactly the same sense that a character named Count Dracula is "immortal", "brooding", or "fearful" of garlic, crucifixes, and sunlight. Fictional traits we perceive on fictional characters from reading text. Yes, we can set up a script where the narrator periodically re-describes AssistantBot as careful and cautious, and that might help a bit with stopping humans from over-trusting the story they are being read. But trying to ensure logical conclusions arise from cautious reasoning is... well, indirect at best, much like trying to make it better at math by narrating "AssistantBot was good at math and diligent at checking the numbers." > Hallucinating P.S.: "Hallucinations" and prompt-injection are non-ironic examples of "it's not a bug, it's a feature". There's no minor magic incantation that'll permanently banish them without damaging how it all works.
- plaguuuuuu 2y ago
  I'd love to know if the conversational training set includes documents where the AI throws its hands up and goes "actually I have no idea". I'm guessing not.
  - Terr_ 2y ago
    There's also the problem of whether the LLM would learn to generate stories where the AssistantBot gives up in cases that match our own logical reasons, versus ones where the AssistantBot gives up because that's simply what AssistantBots in training-stories usually do when the User character uses words of disagreement and disapproval.
patates 2y ago
Hallucinating is a confidence problem, no? Say, they should be 100% confident that "0.3" follows "0.2 + 0.1 =", but a lot of floating point examples on the internet make them less confident. On a much more nuanced problem, "0.30000000000000004" may get more and more confidence. This is what makes them "hallucinate", did I get it wrong? (in other words, am I hallucinating myself? :) )

Hallucinating is fine but overconfidence is the problem. But I heard it's not an easy problem to solve.

jacksnipe 2y ago

Unfortunately, in the system most of us work in today, I think overconfidence is an intelligent behavior.

onemoresoop 2y ago
I find it extremely dumb to see overconfident people that really have nothing special about them or are even incompetent. These people are not contributing positively to the system, quite on the contrary.
- xwolfi 2y ago
  But we don't work for the system, fundamentally, we work for ourselves, and the system incentivizes us to work for it by aligning our constraints: if you work that direction, you'll get that reward. Overconfident people ofc do not contribute positively to the system, but they skew the system reward's calculation towards them: I swear I've done that work in that direction, where's my reward ? In a sense, they are extremely successful: they managed to do very low effort, get very high reward, help themselves like all of us but at a much better profit margin, by sacrificing a system that, let's be honest, none of us care about really. Your problem maybe, is that you swallowed the little BS the system fed you while incentivizing you: that the system matters more than yourself, at least at a greater extent than healthy ? And you see the same thing with AI: these things convince people so deeply of their intelligence that it blew to such proportion that NVidia is now worth trillions. I had a colleague mumbling yesterday that his wife now speaks more with ChatGPT than him. Overconfidence is a positive attribute... for oneself.
  - onemoresoop 2y ago
    Not talking about THE system or any system in particular same way gaming the system doesn’t refer to any system but just cheating in general. And if you like overconfident people good for you, I can’t stand them because they’re hollow with no basis in reality, flawed like everyone, just pumping out their egos with hot air. And your reasoning that overconfidence is a positive attribute doesn’t make much sense to me but we’re entitled to our own opinions.
  - butlike 2y ago
    Overconfident people are conquerors. Conquerors do not contribute positively to a harmonious system, true, but I'm not so sure we can glean the system is supposed to be harmonious. If one contributes "positively" to the system, everyone's value increases and the solution becomes more homogenized. Once the system is homogenized enough, it becomes vulnerable to adversity from an outside force. If the system is not harmonious/non-homoginized, the attacker would be drawn to the most powerful point in the system. Overconfident people aren't evil, they're simply stressing the system to make sure it can handle adversity from an outside force. They're saying: "listen, I'm going to take what you have, and you should be so happy that's all I'm taking." So I think overconfidence is a positive attribute for the system as well as for the overconfident individual. It's not a positive attribute for the local parties getting run over by the overconfident individual.
  - jacksnipe 2y ago
    Yeah this is what I meant, both in the behavior being intelligent and it being unfortunate that this is the case. It’d be nice if the most self-maximizing behavior were also the best behavior for the global system, but it doesn’t seem that it is.
- tomComb 2y ago
  But being like that can get you elected.
  - Freedom2 2y ago
    In what country?
    - tbossanova 2y ago
      Quite a few :(
    - chromehearts 2y ago
      All of them
  - sim7c00 2y ago
    and getting yourself elected while being underqualified is intelligent? i think its not. its stupid and damaging behavior based in selfish desires. about as far from intelligent you can get.
    - rcxdude 2y ago
      Intelligence is seperate from goals: if you're only interested in gaining power and wealth for yourself, then concern about the rest of the system is only incidental to what you can get for yourself.
  - throw4847285 2y ago
    That's a fallacy. There are certainly some unqualified elected leaders, but humans living in democratic societies have yet to shake the mental framework we've constructed from centuries without self-rule. We invest way more authority into a single executive than they actually have, and blame them for everything that goes wrong despite the fact that modern democracies are hugely complex systems in which authority is distributed across numerous people. When the government fails to meet people's needs, they lack the capacity to point at a particular Senator or a small party in a ruling coalition and blame them. It's always the executive. Of course, the result is that people get fed up and decide that the problem has been not that democratic societies are hard to govern by design (they have to reflect the disparate desires of countless people) but that the executive was too weak. They get behind whatever candidate is charismatic enough to convince them that they will govern the way the people already thought the previous executives were governing, just badly. The result is an incompetent tyrant.
- butlike 2y ago
  Overconfidence allows you to act, increasing survivability. Thinking is a "weak" trait.

Terr_ 2y ago

> overconfidence is the problem. The problem is a bit deeper than that, because what we perceive as "confidence" is itself also an illusion. The (real) algorithm takes documents and makes them longer, and some humans configured a document that looks like a conversation between "User" and "AssistantBot", and they also wrote some code to act-out things that look like dialogue for one of the characters. The (real) trait of confidence involves next-token statistics. In contrast, the character named AssistantBot is "overconfident" in exactly the same sense that a character named Count Dracula is "immortal", "brooding", or "fearful" of garlic, crucifixes, and sunlight. Fictional traits we perceive on fictional characters from reading text. Yes, we can set up a script where the narrator periodically re-describes AssistantBot as careful and cautious, and that might help a bit with stopping humans from over-trusting the story they are being read. But trying to ensure logical conclusions arise from cautious reasoning is... well, indirect at best, much like trying to make it better at math by narrating "AssistantBot was good at math and diligent at checking the numbers." > Hallucinating P.S.: "Hallucinations" and prompt-injection are non-ironic examples of "it's not a bug, it's a feature". There's no minor magic incantation that'll permanently banish them without damaging how it all works.

plaguuuuuu 2y ago
I'd love to know if the conversational training set includes documents where the AI throws its hands up and goes "actually I have no idea". I'm guessing not.
- Terr_ 2y ago
  There's also the problem of whether the LLM would learn to generate stories where the AssistantBot gives up in cases that match our own logical reasons, versus ones where the AssistantBot gives up because that's simply what AssistantBots in training-stories usually do when the User character uses words of disagreement and disapproval.

patates 2y ago

Hallucinating is a confidence problem, no? Say, they should be 100% confident that "0.3" follows "0.2 + 0.1 =", but a lot of floating point examples on the internet make them less confident. On a much more nuanced problem, "0.30000000000000004" may get more and more confidence. This is what makes them "hallucinate", did I get it wrong? (in other words, am I hallucinating myself? :) )