4 ms·

This is, once again, about diversity and the lack thereof (and I don't mean diversity in a political sense). LLMs seem fundamentally incapable of producing tru

This is, once again, about diversity and the lack thereof (and I don't mean diversity in a political sense). LLMs seem fundamentally incapable of producing truly diverse outputs, truly creative and different responses to the same prompts in different runs. Because you and me use the same Claude, if you want a website and I want a website, we'll get (almost) the same website. This is not some BS about "the average of its training data", most of the LLM style (both in design and in text) comes from reinforcement learning. You could RL Claude to produce a very different style, but you couldn't RL it to produce a different style for me than it does for you. I think this is also where a lot of the complaints about "Claude writing" come from.

CuriouslyC 15d ago
RL causes distributional collapse, it's how the models get consistent. Anyone who generated images with early gen (SD1.5-2) models will remember the wild variance between seeds, which newer models have mostly lost, and similarly GPT3.5/4 could produce weirder, more original outputs even if they were less consistently "good" in some sense. It's worth mentioning that they do RL for aesthetics to some degree based on human expert feedback, but whatever the model tends to produce quickly becomes debased by its ubiquity. They could RL for output diversity, but it's less well studied and likely to cause minor regressions in coding performance, at least until the algorithms are dialed in.
- miki123211 13d ago
  Yes! There was a great paper at Neurips 2025, where they showed that all "capabilities" that models get via RLVR were in fact already there, if you did pass@k with a sufficiently high k. RLVR makes the models consistently use tricks that tend to work and bring high rewards, which makes them better when k is low, but it suppresses the unusual, which is actually worse when k is high.

CuriouslyC 15d ago

RL causes distributional collapse, it's how the models get consistent. Anyone who generated images with early gen (SD1.5-2) models will remember the wild variance between seeds, which newer models have mostly lost, and similarly GPT3.5/4 could produce weirder, more original outputs even if they were less consistently "good" in some sense. It's worth mentioning that they do RL for aesthetics to some degree based on human expert feedback, but whatever the model tends to produce quickly becomes debased by its ubiquity. They could RL for output diversity, but it's less well studied and likely to cause minor regressions in coding performance, at least until the algorithms are dialed in.

miki123211 13d ago
Yes! There was a great paper at Neurips 2025, where they showed that all "capabilities" that models get via RLVR were in fact already there, if you did pass@k with a sufficiently high k. RLVR makes the models consistently use tricks that tend to work and bring high rewards, which makes them better when k is low, but it suppresses the unusual, which is actually worse when k is high.