5 ms·
Prover-Verifier Games improve legibility of language model outputs
- rjeli 2y agoFunny that when I reached the "Key Findings" section, my brain immediately parsed it as ChatGPT output. Maybe it's the bullet points, the word choice, or just the font...
- Der_Einzige 2y agoThere appears to be a coherent effort among the general populace , conscious or unconscious, to shape discourse going forward to look more ChatGPT style in general. Words like “delve”, “crucial” etc have become more common even among real people in face to face communication and in record time. Much as I find it overly formal, I support it on the grounds that it frustrates attempts to “detect” if LLMs are used and that is very good.
- pfdietz 2y agoWell, as long as they don't delve too greedily and too deep.
- drdeca 2y ago> it frustrates attempts to “detect” if LLMs are used and that is very good. Why is that good?
- Der_Einzige 2y agoIf you are asking, you’re the kind of person it’s designed to frustrate. Good. Stay frustrated.
- drdeca 2y agoIs this an insult, or a criticism? If you think I’m doing something I shouldn’t, tell me what you think I’m doing that I shouldn’t, and why I shouldn’t? Why would you just wish me ill?
- molave 2y agoI can tell technical papers influenced ChatGPT's outputs the most. Most of the articles generated using it may be regurgitated, but I can't deny how easily digestable the info is when presented that way.
- c3534l 2y agoIt seems like a lot of people these days are doing generative adversarial AI and the pretending like they invented a new thing.
- HanClinto 2y agoGANs have been used for a long time in order to improve the training of images -- it seems like we're finally starting to see this approach catch on for LLMs. I'm aware of the SPAG paper -- who else have you seen take this approach with LLMs lately? https://github.com/Linear95/SPAG https://github.com/Linear95/SPAG
- nilamo 2y agoI was thinking the same thing. GANs aren't new, but it's cool that we're using them in new ways.
- HanClinto 2y agoBeautiful! OpenAI isn't just training a model to produce more-verifiable correct answers -- it's leveraging an adversarial relationship to train a model that's better at being correct, and also a model that's better at deceiving / being wrong. This is the key. There are three agents here: * A "verifier" (a small model, whose job it is to discern correct answers from incorrect answers) * A "helpful prover" (blue team, whose job it is to produce correct answers with an easy-to-follow explanation) * A "sneaky prover" (red team, whose job it is to produce incorrect answers with a deceptive explanation) By arranging these three models in an adversarial relationship with a true reinforcement learning feedback loop, the entire model grows and gets better. This is fantastic to read, and corroborates the results achieved by SPAG -- easily one of my favorite papers from the past year. SPAG pioneered (as far as I'm aware) the approach of using adversarial language games in a true reinforcement-learning setup (not merely RLHF, which isn't true RL), and showed that training models in adversarial language games can show generalized improvements even in areas not directly related to the game. [1] Ever since the SPAG paper came out, I've been daydreaming about the different sorts of adversarial games that one could use to train LLMs. I've written down a bunch of notes on the subject [2] (in case anyone else is reading my rambling notes). I would really like to see some of these experiments actually get up and running on open-source LLMs -- I'm excited to see if / how they could be used to improve the quality of some of the open-source base models that are floating around out there. [1] https://github.com/Linear95/SPAG https://github.com/Linear95/SPAG [2] https://github.com/HanClinto/MENTAT https://github.com/HanClinto/MENTAT
- HanClinto 2y agoBecause the Red and Blue agents are both trying to convince a smaller language model of the rightness of their answer, they each have to simplify their logic and wording down. This feels like the ML equivalent of the old adage "If you can't explain it to a six year old, you don't understand it yourself."
- enthulhusiastic 2y agoELI6 why SPAG is better than just the default pretraining method (token context statistics?) of an LLM.
- deleted 2y ago[deleted]
- jkljl 2y ago[dead]
- michwilinski 2y agointeresting, but I don't agree that if we see the "token reasoning" chain it somehow explains how the model got the final answer. what if we trained deceiver models that would provide a sound chain of explanation but then perform some kind of deception and output an incorrect answer? for me personally, explainability has to show how the answer arose from the model mechanics, not sequential model outputs
- HanClinto 2y ago> what if we trained deceiver models that would provide a sound chain of explanation but then perform some kind of deception and output an incorrect answer? You're right on target! That's exactly what they're doing in the paper. They train three models -- a verifier (that rates answers as sounding correct or sounding wrong), a "helpful prover" (that provides correct answers), and "sneaky prover" (that provides incorrect answers that attempt to deceive the verifier into scoring its answer highly). This adversarial relationship between the "helpful prover" and the "sneaky prover" is the cool part of the paper (IMO).
- tfhfj 2y ago[flagged]