5 ms·

Beautiful! OpenAI isn't just training a model to produce more-verifiable correct answers -- it's leveraging an adversarial relationship to train a model that's

Beautiful! OpenAI isn't just training a model to produce more-verifiable correct answers -- it's leveraging an adversarial relationship to train a model that's better at being correct, and also a model that's better at deceiving / being wrong. This is the key. There are three agents here: * A "verifier" (a small model, whose job it is to discern correct answers from incorrect answers) * A "helpful prover" (blue team, whose job it is to produce correct answers with an easy-to-follow explanation) * A "sneaky prover" (red team, whose job it is to produce incorrect answers with a deceptive explanation) By arranging these three models in an adversarial relationship with a true reinforcement learning feedback loop, the entire model grows and gets better. This is fantastic to read, and corroborates the results achieved by SPAG -- easily one of my favorite papers from the past year. SPAG pioneered (as far as I'm aware) the approach of using adversarial language games in a true reinforcement-learning setup (not merely RLHF, which isn't true RL), and showed that training models in adversarial language games can show generalized improvements even in areas not directly related to the game. [1] Ever since the SPAG paper came out, I've been daydreaming about the different sorts of adversarial games that one could use to train LLMs. I've written down a bunch of notes on the subject [2] (in case anyone else is reading my rambling notes). I would really like to see some of these experiments actually get up and running on open-source LLMs -- I'm excited to see if / how they could be used to improve the quality of some of the open-source base models that are floating around out there. [1] https://github.com/Linear95/SPAG https://github.com/Linear95/SPAG [2] https://github.com/HanClinto/MENTAT https://github.com/HanClinto/MENTAT

HanClinto 2y ago
Because the Red and Blue agents are both trying to convince a smaller language model of the rightness of their answer, they each have to simplify their logic and wording down. This feels like the ML equivalent of the old adage "If you can't explain it to a six year old, you don't understand it yourself."
- enthulhusiastic 2y ago
  ELI6 why SPAG is better than just the default pretraining method (token context statistics?) of an LLM.
  - TimPC 2y ago
    The red and blue agents are effectively unlimited sources of true and false examples so you can get far more efficient scale than you can by pre training with labelled inputs. It’s also far more targeted on correct/incorrect rather than a notion of answer quality which doesn’t directly get at hallucination vs reality.
    - blueblaze0 2y ago
      This is impressive, but what prevents the blue agent from generating an incorrect proof of a "true example"? What prevents the red agent from generating a correct disproof of a "false example"? I'm curious how they managed to generate a truly unlimited source of correctly labeled examples.
      - HanClinto 2y ago
        > "but what prevents the blue agent from generating an incorrect proof of a "true example"? That's the role of the Verifier. It's not going to be perfect, and I'm sure some incorrect proofs of true examples slip through, but it's good enough to increase the quality of the model overall. > "What prevents the red agent from generating a correct disproof of a "false example"? And on the other side, it's counterbalanced by the rules engine (math) that can determine absolutely whether or not the right answer is given at the end. The Red and the Blue agents are held in check by the tension between the math engine and the verifier, and they are free to fight back-and-forth within those parameters as long as they are able. Eventually, I think the Red agent loses the ability to attack effectively, and so that's the big limit on OpenAI's arrangement. This particular game isn't balanced enough for this training loop to continue infinitely.
    - Natsu 2y ago
      But how do we know the answer you gave us wasn't generated by the sneaky prover? :)
jgalt212 2y ago
Isn't this exactly how Alpha Go learns and works so good? It always knows the right answer because it knows the rules of the game and can easily compute W-L record. In life, it's hard and very expensive to codify the rules, and compute W-L record.
- HanClinto 2y ago
  Yes, exactly. Using traditional RL is easiest when you're using a landscape with clearly defined rules -- like Go, or Starcraft, or whatever. The trouble is those games don't translate well to other domains -- it can learn about risk and reward and whatnot from Chess, but it can't become a better chatbot. But if the game space can operate through the realm of language and semantics, then the hope is that we can tap into the adversarial growth curve, but for LLMs. As you note, this only works for situations where we can clearly say "winner" or "loser". In OpenAI's case, they use correctness of the math problem as one W/L metric (discrete and measurable) as well as whether the Verifier was able to correctly identify the answer as correct (thus the understandability of the answer is also discrete and measurable). In the SPAG paper, they chose the game of "Taboo" as a way to discretely measure W/L (asking: "did the defender say the secret word or not"). As you noted, it's hard and expensive to codify the rules of life. How do we objectively determine whether one poem is more beautiful than another? I think we're a long way from that. The breakthrough that the SPAG paper showed is that -- by teaching the models to be better at games that involve language and semantics -- that they get better at language-oriented tasks _overall_. And that possibility excites me. Sadly, as I've read further into the paper released by OpenAI, it doesn't appear that adversarial training for explainability increased the accuracy of the model -- and while it was more understandable / verifiable, it wasn't any better. I think a very interesting metric would be to measure the accuracy of the fine-tuned models on unrelated tasks to see if the lessons learned to be better at explaining math problems would help the model perform better for explaining other problems (such as logic or reasoning).
  - bravura 2y ago
    Thank you for the SPAG paper. Do you know how to play questions? https://www.youtube.com/watch?v=u3xIs0aajN4 https://www.youtube.com/watch?v=u3xIs0aajN4 (Tom Stoppard, Rosencrantz and Guildenstern Are Dead). The important question in the OpenAI work that you haven't touched on is how to evaluate superintelligence. I guess I would frame the problem like this: Let's say there is a very esoteric but important branch of abstract mathematics that only a few people claim to understand. Is there a way for us to determine which mathematicians are actually intelligent, and which are bluffing? How?
skdotdan 2y ago
What do you mean by “true” RL?
- HanClinto 2y ago
  True RL is not limited by being tethered to human-annotated data, and it is able to create novel approaches to solve problems. True RL requires a very clear objective function (such as the rules of Go, or Starcraft, or Taboo!) that the model can evaluate itself against. Andrej Karpathy talks about the difference between RLHF and "true" RL here: https://www.youtube.com/watch?v=c3b-JASoPi0&t=1618s https://www.youtube.com/watch?v=c3b-JASoPi0&t=1618s > The other thing is that we're doing reinforcement learning from human feedback (RLHF), but that's like a super weak form of reinforcement learning. I think... what is the equivalent in AlphaGo for RLHF? What is the reward model? What I call it is a "vibe check". Imagine if you wanted to train an AlphaGo RLHF, it would be giving two people two boards and asking: "Which one do you prefer?" -- and then you would take those labels and you would train the model and then you would RL against that. What are the issues with that? It's like, number one -- that's just vibes of the board. That's what you're training against. Number two, if it's a reward model that's a neural net, then it's very easy to overfit to that reward model for the model you're optimizing over, and it's going to find all these spurious ways of hacking that massive model is the problem. > AlphaGo gets around these problems because they have a very clear objective function, and you can RL against it. > So RLHF is nowhere near [true] RL -- it's silly. And the other thing is that imitation is super-silly. RLHF is a nice improvement, but it's still silly, and I think people need to look for better ways of training these models so that it's in the loop with itself and its own psychology, and I think there will probably be unlocks in that direction. In contrast, something like true RL would look like the Multi-Agent Hide-And-Seek training loop: https://www.youtube.com/watch?v=kopoLzvh5jY https://www.youtube.com/watch?v=kopoLzvh5jY
vinnyvichy 2y ago
Some may be reminded of the Magi supercomputers in NERV, but here's a mnemonic inspired by the precogs in Minority Report: 1) helpful prover : the good twin 2) sneaky prover : the evil twin 3) verifier : the foster sister

HanClinto 2y ago

Because the Red and Blue agents are both trying to convince a smaller language model of the rightness of their answer, they each have to simplify their logic and wording down. This feels like the ML equivalent of the old adage "If you can't explain it to a six year old, you don't understand it yourself."

enthulhusiastic 2y ago
ELI6 why SPAG is better than just the default pretraining method (token context statistics?) of an LLM.
- TimPC 2y ago
  The red and blue agents are effectively unlimited sources of true and false examples so you can get far more efficient scale than you can by pre training with labelled inputs. It’s also far more targeted on correct/incorrect rather than a notion of answer quality which doesn’t directly get at hallucination vs reality.
  - blueblaze0 2y ago
    This is impressive, but what prevents the blue agent from generating an incorrect proof of a "true example"? What prevents the red agent from generating a correct disproof of a "false example"? I'm curious how they managed to generate a truly unlimited source of correctly labeled examples.
    - HanClinto 2y ago
      > "but what prevents the blue agent from generating an incorrect proof of a "true example"? That's the role of the Verifier. It's not going to be perfect, and I'm sure some incorrect proofs of true examples slip through, but it's good enough to increase the quality of the model overall. > "What prevents the red agent from generating a correct disproof of a "false example"? And on the other side, it's counterbalanced by the rules engine (math) that can determine absolutely whether or not the right answer is given at the end. The Red and the Blue agents are held in check by the tension between the math engine and the verifier, and they are free to fight back-and-forth within those parameters as long as they are able. Eventually, I think the Red agent loses the ability to attack effectively, and so that's the big limit on OpenAI's arrangement. This particular game isn't balanced enough for this training loop to continue infinitely.
  - Natsu 2y ago
    But how do we know the answer you gave us wasn't generated by the sneaky prover? :)

jgalt212 2y ago

Isn't this exactly how Alpha Go learns and works so good? It always knows the right answer because it knows the rules of the game and can easily compute W-L record. In life, it's hard and very expensive to codify the rules, and compute W-L record.

HanClinto 2y ago
Yes, exactly. Using traditional RL is easiest when you're using a landscape with clearly defined rules -- like Go, or Starcraft, or whatever. The trouble is those games don't translate well to other domains -- it can learn about risk and reward and whatnot from Chess, but it can't become a better chatbot. But if the game space can operate through the realm of language and semantics, then the hope is that we can tap into the adversarial growth curve, but for LLMs. As you note, this only works for situations where we can clearly say "winner" or "loser". In OpenAI's case, they use correctness of the math problem as one W/L metric (discrete and measurable) as well as whether the Verifier was able to correctly identify the answer as correct (thus the understandability of the answer is also discrete and measurable). In the SPAG paper, they chose the game of "Taboo" as a way to discretely measure W/L (asking: "did the defender say the secret word or not"). As you noted, it's hard and expensive to codify the rules of life. How do we objectively determine whether one poem is more beautiful than another? I think we're a long way from that. The breakthrough that the SPAG paper showed is that -- by teaching the models to be better at games that involve language and semantics -- that they get better at language-oriented tasks _overall_. And that possibility excites me. Sadly, as I've read further into the paper released by OpenAI, it doesn't appear that adversarial training for explainability increased the accuracy of the model -- and while it was more understandable / verifiable, it wasn't any better. I think a very interesting metric would be to measure the accuracy of the fine-tuned models on unrelated tasks to see if the lessons learned to be better at explaining math problems would help the model perform better for explaining other problems (such as logic or reasoning).
- bravura 2y ago
  Thank you for the SPAG paper. Do you know how to play questions? https://www.youtube.com/watch?v=u3xIs0aajN4 https://www.youtube.com/watch?v=u3xIs0aajN4 (Tom Stoppard, Rosencrantz and Guildenstern Are Dead). The important question in the OpenAI work that you haven't touched on is how to evaluate superintelligence. I guess I would frame the problem like this: Let's say there is a very esoteric but important branch of abstract mathematics that only a few people claim to understand. Is there a way for us to determine which mathematicians are actually intelligent, and which are bluffing? How?

skdotdan 2y ago

What do you mean by “true” RL?

HanClinto 2y ago
True RL is not limited by being tethered to human-annotated data, and it is able to create novel approaches to solve problems. True RL requires a very clear objective function (such as the rules of Go, or Starcraft, or Taboo!) that the model can evaluate itself against. Andrej Karpathy talks about the difference between RLHF and "true" RL here: https://www.youtube.com/watch?v=c3b-JASoPi0&t=1618s https://www.youtube.com/watch?v=c3b-JASoPi0&t=1618s > The other thing is that we're doing reinforcement learning from human feedback (RLHF), but that's like a super weak form of reinforcement learning. I think... what is the equivalent in AlphaGo for RLHF? What is the reward model? What I call it is a "vibe check". Imagine if you wanted to train an AlphaGo RLHF, it would be giving two people two boards and asking: "Which one do you prefer?" -- and then you would take those labels and you would train the model and then you would RL against that. What are the issues with that? It's like, number one -- that's just vibes of the board. That's what you're training against. Number two, if it's a reward model that's a neural net, then it's very easy to overfit to that reward model for the model you're optimizing over, and it's going to find all these spurious ways of hacking that massive model is the problem. > AlphaGo gets around these problems because they have a very clear objective function, and you can RL against it. > So RLHF is nowhere near [true] RL -- it's silly. And the other thing is that imitation is super-silly. RLHF is a nice improvement, but it's still silly, and I think people need to look for better ways of training these models so that it's in the loop with itself and its own psychology, and I think there will probably be unlocks in that direction. In contrast, something like true RL would look like the Multi-Agent Hide-And-Seek training loop: https://www.youtube.com/watch?v=kopoLzvh5jY https://www.youtube.com/watch?v=kopoLzvh5jY

vinnyvichy 2y ago

Some may be reminded of the Magi supercomputers in NERV, but here's a mnemonic inspired by the precogs in Minority Report: 1) helpful prover : the good twin 2) sneaky prover : the evil twin 3) verifier : the foster sister