3 ms·

Maybe because LLMs are literally designed to generate boring content. They are trained to output the best possible prediction - i.e most predictable version - o

Maybe because LLMs are literally designed to generate boring content. They are trained to output the best possible prediction - i.e most predictable version - of how the input will continue (i.e the least surprising/interesting continuation). Humans use surprise (= prediction failure) as a learning signal and to focus our attention, so not surprising that we dose off and lose interest when something is highly predictable. Good human authors know how to introduce plot twists and suspense, to keep the reader engaged and guessing. Imagine an entire novel that was instead written by an LLM, where the only plot twists are the predictable ones it copied from it's training material. The same goes for any LLM generated prose, whether advertising copy or training manual - it's going to be a snooze fest because it's so predictable - very low signal to noise ratio.

deleted 1y ago
[deleted]
nullc 1y ago
Base models are less boring, a lot less-- so I think it's less a product of most likely prediction, I think it's more of a product of reinforcement which biases towards text that uses a lot of words but says little, since saying nothing is at least not saying something identifiably hallucinatory.
- HarHarVeryFunny 1y ago
  RL isn't in general going to bias the model to use a lot of words, unless your RL training had the goal of favoring long over short responses. There are now multiple levels of RL being used to post-train these models, from RLHF (use RL to bias the model to generate outputs matching human feedback preferences) to RL used to improve reasoning by generating reasoning steps that lead to verified correct conclusions (in areas like math and programming where correctness can be verified). RLHF (not RL in general) may lead to longer more verbose outputs to the extent that human testers indicated longer responses as their preference. Maybe testers are easily bullshitted and like something that is longer and sounds like a more comprehensive authoritative answer ? There is also the fact that an LLM is, unless prompted otherwise, is trying to predict Mr. Average (entire training set), who is more likely to waffle on than an expert who will cut to the chase and just give the facts, which they have a firm grip of. You can of course prompt the model to behave like an expert, or any given role, or to be more concise, which may or may not result in better output. It's a bit like asking the model to summarize when it's not really summarizing but instead predicting what a summary would look like (form vs function).
EnPissant 1y ago
But they predict based on context. If you prompt it with "You are a creative writing bot that excels in generating unpredictable, yet coherent text", does it predict what is unpredictable?
- HarHarVeryFunny 1y ago
  Yes, that's all it can ever do, predict a continuation. It's not a person though, so it'd not be predicting what you might find unpredictable, but rather predicting training samples that it associated with unpredictability (e.g. with some following feedback of "well, that was unpredictable!"). IOW it'll be predicting the most predictable unpredictability that it can.

deleted 1y ago

[deleted]

nullc 1y ago

Base models are less boring, a lot less-- so I think it's less a product of most likely prediction, I think it's more of a product of reinforcement which biases towards text that uses a lot of words but says little, since saying nothing is at least not saying something identifiably hallucinatory.

HarHarVeryFunny 1y ago
RL isn't in general going to bias the model to use a lot of words, unless your RL training had the goal of favoring long over short responses. There are now multiple levels of RL being used to post-train these models, from RLHF (use RL to bias the model to generate outputs matching human feedback preferences) to RL used to improve reasoning by generating reasoning steps that lead to verified correct conclusions (in areas like math and programming where correctness can be verified). RLHF (not RL in general) may lead to longer more verbose outputs to the extent that human testers indicated longer responses as their preference. Maybe testers are easily bullshitted and like something that is longer and sounds like a more comprehensive authoritative answer ? There is also the fact that an LLM is, unless prompted otherwise, is trying to predict Mr. Average (entire training set), who is more likely to waffle on than an expert who will cut to the chase and just give the facts, which they have a firm grip of. You can of course prompt the model to behave like an expert, or any given role, or to be more concise, which may or may not result in better output. It's a bit like asking the model to summarize when it's not really summarizing but instead predicting what a summary would look like (form vs function).

EnPissant 1y ago

But they predict based on context. If you prompt it with "You are a creative writing bot that excels in generating unpredictable, yet coherent text", does it predict what is unpredictable?

HarHarVeryFunny 1y ago
Yes, that's all it can ever do, predict a continuation. It's not a person though, so it'd not be predicting what you might find unpredictable, but rather predicting training samples that it associated with unpredictability (e.g. with some following feedback of "well, that was unpredictable!"). IOW it'll be predicting the most predictable unpredictability that it can.