4 ms·
Sorry if this is a dumb question, but how does that make sure that the training process is not going into the wrong direction because of error accumulation? Ma
by starbugs 3y ago
Sorry if this is a dumb question, but how does that make sure that the training process is not going into the wrong direction because of error accumulation?
Maybe I didn't understand something fundamental here. (Not an LLM expert.)
- deleted 3y ago[deleted]
- candiodari 3y agoIt doesn't. I would like to add, there's plenty of examples, some in math (e.g. geometry) playing out over >1000 years and dozens of generations, of the same happening in humans. That said, for both humans and this kind of LLMs, it does appear to improve performance, certainly in the near term.
- starbugs 3y agoCertainly there’s this problem in many areas. I was just wondering how big of a deal that might be in this case. Just had another of those experiences with GPT4 where it goes into a contradictory loop it cannot recover from. It seems there might be a big difference between long term cycles and short term severe degradation as in inbreeding and this paper‘s abstract sounded a bit like that to me. If the results indicate improved performance, then it doesn’t seem to be that big of a deal (yet?). Thanks for the explanation!
- huac 3y agoI don't think it does. And there is a pretty big risk that you end up picking up on some quirk ("bias") of your reward model that doesn't reflect reality -- GPT4 preferring longer answers is one such commonly observed bias. AFAIK there is not a great theoretical basis for why we can avoid mode collapse, except empirically the models are good enough to survive some bootstrapping.