3 ms·

Hyperbolic article from writers that didn’t understand the papers. The actual problem is that LLMs don’t produce material with the same statistical distributio

Hyperbolic article from writers that didn’t understand the papers. The actual problem is that LLMs don’t produce material with the same statistical distribution as their training sets, which is obvious if you think about it. For instance, the GPT training set doubtless includes plenty of swearing and slurs, but its output does not (even if you prompt engineer your way to an example, they will still be less represented in the total output). Same is true for the long tail. If I ask GPT to complete “oregano tastes like”, there are lots of fairly likely completions, but it’s probably never going to produce “tires”, even though exceptionally rare combos probably occur in the training data. I’m not sure this is a bad thing. Maybe? Is there some productivity or moral imperative to leave the distribution of words/concepts the way it was before LLMs?

HarHarVeryFunny 3y ago
> The actual problem is that LLMs don’t produce material with the same statistical distribution as their training sets Yes, and that seems pretty much unavoidable even without SFT and RLHF "filtering"... There's no obvious way of recreating the "long tail" distribution of the overall training set. Turning the sampling temperature up isn't the same since that'll affect the entire generated output, while a human might have chosen to pop a single spicy word choice into an otherwise boilerplate sentence. One way to get more human-like variety of these models would be to condition them on specific writers with different styles, but of course in practice certain styles would be used more often than others and a corpora of machine-generated text would therefore not have same statistics as one of (more randomly chosen) human text, even if it did have more variety than would otherwise be the case. Still, conditioning on a specific writing style might be a future direction. Would be useful for people wanting to generate text in their own personal style rather than just asking the LLM to adopt some generic persona.

HarHarVeryFunny 3y ago

> The actual problem is that LLMs don’t produce material with the same statistical distribution as their training sets Yes, and that seems pretty much unavoidable even without SFT and RLHF "filtering"... There's no obvious way of recreating the "long tail" distribution of the overall training set. Turning the sampling temperature up isn't the same since that'll affect the entire generated output, while a human might have chosen to pop a single spicy word choice into an otherwise boilerplate sentence. One way to get more human-like variety of these models would be to condition them on specific writers with different styles, but of course in practice certain styles would be used more often than others and a corpora of machine-generated text would therefore not have same statistics as one of (more randomly chosen) human text, even if it did have more variety than would otherwise be the case. Still, conditioning on a specific writing style might be a future direction. Would be useful for people wanting to generate text in their own personal style rather than just asking the LLM to adopt some generic persona.