5 ms·

> LLMs are generative models trained to produce text similar to the text on which they were trained (or for the pedantic: minimize perplexity). They are not tra

> LLMs are generative models trained to produce text similar to the text on which they were trained (or for the pedantic: minimize perplexity). They are not trained to output facts or truths or any other specific kind of text. If you train them on text that is largely true, then the output is also largely true. To ignore this is to miss the point of LLMs altogether.

giantrobot 3y ago
But then companies go train them on "The Internet". This includes not just factual (or mostly factual) content like Wikipedia but also SEO spam sites, YouTube comment sections, literal fake news articles, and fan fiction. Even if they're limited to "mostly true" content they'll happily and confidently confabulate "facts". They'll make syntactically and grammatically correct sentences that are actually authentic frontier gibberish.
- JohnFen 3y ago
  > But then companies go train them on "The Internet". This is the bit that I think a lot of people miss. If the model was trained on stuff from the internet, then what the model will produce will reflect that, and the internet is chock full of bullshit. Same thing about when LLMs produce objectionable responses (racist, violent, whatever). Those are a direct reflection of the nature of the training material.
  - CuriouslyC 3y ago
    Pretraining is "off the internet" but task training/rlhf is not.
swatcoder 3y ago
Have any of the leading models been trained on that kind of corpus? Could a corpus like that even be constructed without vociferous and immediate dispute about what amounts to "largely true"? Would a corpus like that have enough suitable data to generate conversational output that feels satisfying? I thought that most/all of these big models are relying on a much much larger and more diverse body of content. Is that wrong?
- stale2002 3y ago
  > Would a corpus like that have enough suitable data to generate conversational output that feels satisfying? Yes? The point is that we can use LLMs right now. They mostly work. That is the empirical evidence that is immediately verifiable right now. This is not a hypothetical. LLMs mostly work.
  - swatcoder 3y ago
    Huh? I don't understand how this engages with what I wrote or what I was responding to. Yes, it's very very obvious that the conversational LLM's we engagw with mostly work for conversational UX. The discussion is about the above commenter's reference to "largely true" data. I'm not aware of any leading LLM built on something that could be characterized that way, but I'm genuinely curious to hear otherwise.
    - deleted 3y ago
      [deleted]
  - rsynnott 3y ago
    See Tim Bray’s example; the LLM very much did not mostly work once it was talking about something that he knew about; most of what it said was nonsense. I suspect a _huge_ driver of the hype is that people are not usually experts on the things that they ask these about, so they see a magical oracle rather than a nonsense generator.
- CuriouslyC 3y ago
  Phi2 was trained on a very highly controlled corpus.
- kergonath 3y ago
  > Have any of the leading models been trained on that kind of corpus? Does such a corpus even exist? Maybe a bunch of Mathematics papers, and even then... To build a corpus like that would mean trusting the human gatekeepers or curators, which just shifts the problem, because humans are also prone to this sort of mistake.
unshavedyak 3y ago
There's still a missing factor i suspect tho. Given how frequently wrong and hallucination prone humans are, i don't think we're _that_ different in this context. Nonetheless we can somehow inspect our thoughts, and come up with a degree of confidence. But what gives us that ability? I don't trust human thought at all. Police have to be careful what they say as to not pollute the minds of anyone they're questioning - we're just insanely prone to flat out lie without knowing. So to one degree, we seem quite similar to hallucinating LLMs. So what gives us the ability to nonetheless identify "truth" from our memory? Is it an ability to trace back to training data? The less clear the path is, perhaps the more likely we think we don't "know" the answer?
- eddd-ddde 3y ago
  I also feel this way. We are not inherently different to LLMs, we are just better at some things, worse at others. At the end of the day we both are just atoms and electricity as ruled by the laws of physics.

giantrobot 3y ago

But then companies go train them on "The Internet". This includes not just factual (or mostly factual) content like Wikipedia but also SEO spam sites, YouTube comment sections, literal fake news articles, and fan fiction. Even if they're limited to "mostly true" content they'll happily and confidently confabulate "facts". They'll make syntactically and grammatically correct sentences that are actually authentic frontier gibberish.

JohnFen 3y ago
> But then companies go train them on "The Internet". This is the bit that I think a lot of people miss. If the model was trained on stuff from the internet, then what the model will produce will reflect that, and the internet is chock full of bullshit. Same thing about when LLMs produce objectionable responses (racist, violent, whatever). Those are a direct reflection of the nature of the training material.
- CuriouslyC 3y ago
  Pretraining is "off the internet" but task training/rlhf is not.

swatcoder 3y ago

Have any of the leading models been trained on that kind of corpus? Could a corpus like that even be constructed without vociferous and immediate dispute about what amounts to "largely true"? Would a corpus like that have enough suitable data to generate conversational output that feels satisfying? I thought that most/all of these big models are relying on a much much larger and more diverse body of content. Is that wrong?

stale2002 3y ago
> Would a corpus like that have enough suitable data to generate conversational output that feels satisfying? Yes? The point is that we can use LLMs right now. They mostly work. That is the empirical evidence that is immediately verifiable right now. This is not a hypothetical. LLMs mostly work.
- swatcoder 3y ago
  Huh? I don't understand how this engages with what I wrote or what I was responding to. Yes, it's very very obvious that the conversational LLM's we engagw with mostly work for conversational UX. The discussion is about the above commenter's reference to "largely true" data. I'm not aware of any leading LLM built on something that could be characterized that way, but I'm genuinely curious to hear otherwise.
  - deleted 3y ago
    [deleted]
- rsynnott 3y ago
  See Tim Bray’s example; the LLM very much did not mostly work once it was talking about something that he knew about; most of what it said was nonsense. I suspect a _huge_ driver of the hype is that people are not usually experts on the things that they ask these about, so they see a magical oracle rather than a nonsense generator.
CuriouslyC 3y ago
Phi2 was trained on a very highly controlled corpus.
kergonath 3y ago
> Have any of the leading models been trained on that kind of corpus? Does such a corpus even exist? Maybe a bunch of Mathematics papers, and even then... To build a corpus like that would mean trusting the human gatekeepers or curators, which just shifts the problem, because humans are also prone to this sort of mistake.

unshavedyak 3y ago

There's still a missing factor i suspect tho. Given how frequently wrong and hallucination prone humans are, i don't think we're _that_ different in this context. Nonetheless we can somehow inspect our thoughts, and come up with a degree of confidence. But what gives us that ability? I don't trust human thought at all. Police have to be careful what they say as to not pollute the minds of anyone they're questioning - we're just insanely prone to flat out lie without knowing. So to one degree, we seem quite similar to hallucinating LLMs. So what gives us the ability to nonetheless identify "truth" from our memory? Is it an ability to trace back to training data? The less clear the path is, perhaps the more likely we think we don't "know" the answer?

eddd-ddde 3y ago
I also feel this way. We are not inherently different to LLMs, we are just better at some things, worse at others. At the end of the day we both are just atoms and electricity as ruled by the laws of physics.