5 ms·

> their understanding of LLM's is imperfect (as is the case for everyone!), and yet they come up with these statements of absolute certainty. Oh interesting, s

> their understanding of LLM's is imperfect (as is the case for everyone!), and yet they come up with these statements of absolute certainty. Oh interesting, so when they accuse ChatGPT of being dangerous because it can be "confidently inaccurate" it's the pot calling the kettle black

zwaps 4y ago
ChatGPT is trained via reinforcement learning to give answers that - ultimately - human non-experts would judge to be plausible. Answers were not trained to be correct or accurate - hence we should not expect ChatGPT to do well there. Sociologists have found that humans seek plausible answers more than correct answers, especially when trying to make sense of a situation. For that reason, the training objective makes sense for a chatbot. Now, anyone who tries to seel ChatGPT as giving truthful answers is of course doing a disfavor to the model and its training objective. However, anyone also trying to sell the above either as a) Deep insight, or b) inherent limitation of LLMs is indeed a pot calling the kettle black. Going from plausibility to truthfulness is probably not a giant leap - at least not compared to what was achieved in the past three years. There's active research on it, and I am sure a good solution will arise. However, ChatGPT is not that, and it isn't meant to be. It's not the training objective. Making this a huge point is again such a BS strawman as the other things I mentioned before.
- mannykannot 4y ago
  > ChatGPT is trained via reinforcement learning to give answers that - ultimately - human non-experts would judge to be plausible. Ostensibly, it is being trained give replies that are more likely than most to be given by humans. On the reasonable (at least IMHO) assumption that people's speech is biased towards saying things that their target audience might see as plausible, this can be seen as doing what you say in the above quoted. This is not, however, the same as being able to judge the plausibility of the semantic content of those statements, so I remain skeptical that the current methods for training LLMs are capable of creating that ability, which in turn leads me to be skeptical that going from plausible statements to reasonably reliably true statements will be a relatively small step in comparison*. I'm open to being surprised, however, as I already have been by what has been achieved so far. The LLM-does-math paper you mention in your sibling comment looks extremely interesting, and maybe I will change my mind... * At least by a continuation or scaling-up of current methods. Humans may be more likely to say truthful things than falsehoods (at least in some areas and some contexts) but a gap has formed between plausible and truthful LLM productions, and I don't see any particular reason to think more of the same will close it.
  - zwaps 4y ago
    What do you mean by current methods though? ChatGPT is trained very differently method wise than GPT3 itself. Whereas GPT3 was strongly based on sequential likelihood in a large corpus, ChatGPT is trained (i would claim) on producing outputs that are judged as good answer and hence plausible. Which is also why the recent iteration of GPT3 is still better than ChatGPT on some tasks. Of course, a truthful model needs a new approach, and perhaps a new baseline model. For instance, starting with code rather than text has been beneficial for recent GPT models, presumably by learning stronger reasoning compositionality. Perhaps, some such baseline on truthfulness needs to be another starting point before large corpora language modeling. Nevertheless, it seems to me this paradigm is more likely to be the way forward than any symbolic approach, say, which is not yet able to even produce text afaik. I mean, even if we need a retrieval style fact checking component, I do now believe there’ll be a transformer equivalent LLM in any future model.
    - mannykannot 4y ago
      Unless the difference in training of GPT-3 and ChatGPT has led to an improvement in the veracity of the replies given, I think the differences in their training are moot in regard to the question of what it will take to get substantially true responses from them. For the reasons I gave in my previous post, I don't regard progress in appearing plausible as necessarily progress towards being able to tell truth from falsehoods. To be clear, I am not claiming that a symbolic approach would be better. Pointing out the shortcomings of current methods does not provide evidence that symbolic approaches will succeed (or vice-versa, for that matter.) This leaves code. On reading your statement about starting with it, it suddenly occurred to me that code is a constrained environment compared to the whole of human discourse. It is obvious, I think, that programming languages are extremely limited in what they can express in comparison to human languages, but perhaps less obviously (at least to me), it seems to follow that what humans can say about programs is also constrained (again, in comparison to human discourse in general - but not nearly as constrained as what can be said in the programming languages themselves.) This leads me to agree with you in this respect (at least tentatively), and here's why: Currently, on Earth, we have one species whose individuals have a well-developed sense of themselves as agents in that world, a similarly well-developed theory of mind about others, and are adroit language users. A few other species have some of these abilities to a limited extent, but not nearly so well developed that they can fully make use of them. As an evolutionist, I suppose that some of our ancestor and related species had intermediate levels of these capabilities, and if we can make AIs that occupy that space we would be making progress. In this view, attempting to match or surpass human performance in language use and understanding the world is about as difficult a target as we could have. As an intermediate target, the domain of program code and the things people say about it has some things going for it: not only is it more constrained (as argued above), but quite a lot has been written about it and is available for training. It is also (arguably) less likely than human language in general to contain statements intended to influence opinions without regard to the truth.

zwaps 4y ago

ChatGPT is trained via reinforcement learning to give answers that - ultimately - human non-experts would judge to be plausible. Answers were not trained to be correct or accurate - hence we should not expect ChatGPT to do well there. Sociologists have found that humans seek plausible answers more than correct answers, especially when trying to make sense of a situation. For that reason, the training objective makes sense for a chatbot. Now, anyone who tries to seel ChatGPT as giving truthful answers is of course doing a disfavor to the model and its training objective. However, anyone also trying to sell the above either as a) Deep insight, or b) inherent limitation of LLMs is indeed a pot calling the kettle black. Going from plausibility to truthfulness is probably not a giant leap - at least not compared to what was achieved in the past three years. There's active research on it, and I am sure a good solution will arise. However, ChatGPT is not that, and it isn't meant to be. It's not the training objective. Making this a huge point is again such a BS strawman as the other things I mentioned before.

mannykannot 4y ago
> ChatGPT is trained via reinforcement learning to give answers that - ultimately - human non-experts would judge to be plausible. Ostensibly, it is being trained give replies that are more likely than most to be given by humans. On the reasonable (at least IMHO) assumption that people's speech is biased towards saying things that their target audience might see as plausible, this can be seen as doing what you say in the above quoted. This is not, however, the same as being able to judge the plausibility of the semantic content of those statements, so I remain skeptical that the current methods for training LLMs are capable of creating that ability, which in turn leads me to be skeptical that going from plausible statements to reasonably reliably true statements will be a relatively small step in comparison*. I'm open to being surprised, however, as I already have been by what has been achieved so far. The LLM-does-math paper you mention in your sibling comment looks extremely interesting, and maybe I will change my mind... * At least by a continuation or scaling-up of current methods. Humans may be more likely to say truthful things than falsehoods (at least in some areas and some contexts) but a gap has formed between plausible and truthful LLM productions, and I don't see any particular reason to think more of the same will close it.
- zwaps 4y ago
  What do you mean by current methods though? ChatGPT is trained very differently method wise than GPT3 itself. Whereas GPT3 was strongly based on sequential likelihood in a large corpus, ChatGPT is trained (i would claim) on producing outputs that are judged as good answer and hence plausible. Which is also why the recent iteration of GPT3 is still better than ChatGPT on some tasks. Of course, a truthful model needs a new approach, and perhaps a new baseline model. For instance, starting with code rather than text has been beneficial for recent GPT models, presumably by learning stronger reasoning compositionality. Perhaps, some such baseline on truthfulness needs to be another starting point before large corpora language modeling. Nevertheless, it seems to me this paradigm is more likely to be the way forward than any symbolic approach, say, which is not yet able to even produce text afaik. I mean, even if we need a retrieval style fact checking component, I do now believe there’ll be a transformer equivalent LLM in any future model.
  - mannykannot 4y ago
    Unless the difference in training of GPT-3 and ChatGPT has led to an improvement in the veracity of the replies given, I think the differences in their training are moot in regard to the question of what it will take to get substantially true responses from them. For the reasons I gave in my previous post, I don't regard progress in appearing plausible as necessarily progress towards being able to tell truth from falsehoods. To be clear, I am not claiming that a symbolic approach would be better. Pointing out the shortcomings of current methods does not provide evidence that symbolic approaches will succeed (or vice-versa, for that matter.) This leaves code. On reading your statement about starting with it, it suddenly occurred to me that code is a constrained environment compared to the whole of human discourse. It is obvious, I think, that programming languages are extremely limited in what they can express in comparison to human languages, but perhaps less obviously (at least to me), it seems to follow that what humans can say about programs is also constrained (again, in comparison to human discourse in general - but not nearly as constrained as what can be said in the programming languages themselves.) This leads me to agree with you in this respect (at least tentatively), and here's why: Currently, on Earth, we have one species whose individuals have a well-developed sense of themselves as agents in that world, a similarly well-developed theory of mind about others, and are adroit language users. A few other species have some of these abilities to a limited extent, but not nearly so well developed that they can fully make use of them. As an evolutionist, I suppose that some of our ancestor and related species had intermediate levels of these capabilities, and if we can make AIs that occupy that space we would be making progress. In this view, attempting to match or surpass human performance in language use and understanding the world is about as difficult a target as we could have. As an intermediate target, the domain of program code and the things people say about it has some things going for it: not only is it more constrained (as argued above), but quite a lot has been written about it and is available for training. It is also (arguably) less likely than human language in general to contain statements intended to influence opinions without regard to the truth.