3 ms·

How does it determine if its generated text is factual?

• simonw 4y ago
  The short answer is it can't. It's arguable whether anyone can - for a human being, determining if text is "factual" can be incredibly difficult. A better answer: if a fact is present many, many times in training data - "Paris is the capital of France" for example, it's much more likely to be true. Also influential: RLHF - Reinforcement Learning from Human Feedback. This is the process by which human labellers rate answers from LLMs - if they consistently rate up "facts" the models have a better chance of outputting factual information, at least if they can relate it to the up-voted responses somehow.
  • flanked-evergl 4y ago
    > The short answer is it can't. It's arguable whether anyone can - for a human being, determining if text is "factual" can be incredibly difficult. Yet, most adults I deal with don't make false things up out of whole cloth as much as ChatGPT does, and it really does not seem like it is that difficult for them. Children do this quite often though, and some adults do, but most don't. > A better answer: if a fact is present many, many times in training data - "Paris is the capital of France" for example, it's much more likely to be true. I think it is quite expected that it is biased to generating output that represent its training data, but this seems like it is not really a solution to the problem. Furthermore, sometimes I want ChatGPT to make things up which is not identical to training data. How do you get it to recognize that it is operating in the realm of fact or not? I'm not sure larger models with more parameters gets you to where you want to go. I think many people overstate the problem, I think it is not that serious, but I think a lot of people also try and just dismiss the issue.