> The actual problem is that LLMs don’t produce material with the same statistical distribution as their training sets Yes, and that seems pretty much unavoida

> The actual problem is that LLMs don’t produce material with the same statistical distribution as their training sets Yes, and that seems pretty much unavoidable even without SFT and RLHF "filtering"... There's no obvious way of recreating the "long tail" distribution of the overall training set. Turning the sampling temperature up isn't the same since that'll affect the entire generated output, while a human might have chosen to pop a single spicy word choice into an otherwise boilerplate sentence. One way to get more human-like variety of these models would be to condition them on specific writers with different styles, but of course in practice certain styles would be used more often than others and a corpora of machine-generated text would therefore not have same statistics as one of (more randomly chosen) human text, even if it did have more variety than would otherwise be the case. Still, conditioning on a specific writing style might be a future direction. Would be useful for people wanting to generate text in their own personal style rather than just asking the LLM to adopt some generic persona.