3 ms·

Yes, exactly. Using traditional RL is easiest when you're using a landscape with clearly defined rules -- like Go, or Starcraft, or whatever. The trouble is th

Yes, exactly. Using traditional RL is easiest when you're using a landscape with clearly defined rules -- like Go, or Starcraft, or whatever. The trouble is those games don't translate well to other domains -- it can learn about risk and reward and whatnot from Chess, but it can't become a better chatbot. But if the game space can operate through the realm of language and semantics, then the hope is that we can tap into the adversarial growth curve, but for LLMs. As you note, this only works for situations where we can clearly say "winner" or "loser". In OpenAI's case, they use correctness of the math problem as one W/L metric (discrete and measurable) as well as whether the Verifier was able to correctly identify the answer as correct (thus the understandability of the answer is also discrete and measurable). In the SPAG paper, they chose the game of "Taboo" as a way to discretely measure W/L (asking: "did the defender say the secret word or not"). As you noted, it's hard and expensive to codify the rules of life. How do we objectively determine whether one poem is more beautiful than another? I think we're a long way from that. The breakthrough that the SPAG paper showed is that -- by teaching the models to be better at games that involve language and semantics -- that they get better at language-oriented tasks _overall_. And that possibility excites me. Sadly, as I've read further into the paper released by OpenAI, it doesn't appear that adversarial training for explainability increased the accuracy of the model -- and while it was more understandable / verifiable, it wasn't any better. I think a very interesting metric would be to measure the accuracy of the fine-tuned models on unrelated tasks to see if the lessons learned to be better at explaining math problems would help the model perform better for explaining other problems (such as logic or reasoning).

bravura 2y ago
Thank you for the SPAG paper. Do you know how to play questions? https://www.youtube.com/watch?v=u3xIs0aajN4 https://www.youtube.com/watch?v=u3xIs0aajN4 (Tom Stoppard, Rosencrantz and Guildenstern Are Dead). The important question in the OpenAI work that you haven't touched on is how to evaluate superintelligence. I guess I would frame the problem like this: Let's say there is a very esoteric but important branch of abstract mathematics that only a few people claim to understand. Is there a way for us to determine which mathematicians are actually intelligent, and which are bluffing? How?
- HanClinto 2y ago
  Oh that was a brilliant video clip. I hadn't seen that before, thank you!! > The important question in the OpenAI work that you haven't touched on is how to evaluate superintelligence. I guess I would frame the problem like this: > Let's say there is a very esoteric but important branch of abstract mathematics that only a few people claim to understand. Is there a way for us to determine which mathematicians are actually intelligent, and which are bluffing? How? This is a tricky one. To my dog, I am revered as a super-being of intelligence and capability. But if he watches me play grandmaster-level chess, or writing a paper on abstract mathematics -- it must look like insanity. In sci-fi, I rather like the image of super-intelligence from one of my favorite short-stories: "When the Yogurt Took Over" [1] > No one argues with the yogurt. No one tweaks its formulas. The rest of the time it rests there in its factory, thinking about whatever intelligent fermented milk thinks about. It just sits there in its vat -- and its actions seem largely incomprehensible to us -- as incomprehensible as me playing Magic: The Gathering is to my dog. It must look like lunacy. (given what I spend on the game, I'm not sure it's not) So if we're going to evaluate superintelligence, then I feel that -- for starters -- it must be on somewhat of a clear playing-field. We can clearly evaluate super-ability in Chess, in Go, and in Starcraft 2 because there are clearly defined rules. The only true test of whether one is superior to another will be because "it works". Until we can test abstract mathematics objectively, then I'm not sure we could ever judge. In so far as questions of particle physics and whatnot could actually be tested -- those feel like the sorts of areas where we might be able to evaluate superintelligence. But SPAG is much smaller than that. The hope that SPAG offers is that -- as long as the game rules leverage things like language and semantics -- then (assuming the model is able to generalize), then the increased mastery of language will transfer to other tasks. And the SPAG results seem to bear that out. [1] https://whatever.scalzi.com/2010/10/02/when-the-yogurt-took-over-a-short-story/ https://whatever.scalzi.com/2010/10/02/when-the-yogurt-took-...

bravura 2y ago

Thank you for the SPAG paper. Do you know how to play questions? https://www.youtube.com/watch?v=u3xIs0aajN4 https://www.youtube.com/watch?v=u3xIs0aajN4 (Tom Stoppard, Rosencrantz and Guildenstern Are Dead). The important question in the OpenAI work that you haven't touched on is how to evaluate superintelligence. I guess I would frame the problem like this: Let's say there is a very esoteric but important branch of abstract mathematics that only a few people claim to understand. Is there a way for us to determine which mathematicians are actually intelligent, and which are bluffing? How?

HanClinto 2y ago
Oh that was a brilliant video clip. I hadn't seen that before, thank you!! > The important question in the OpenAI work that you haven't touched on is how to evaluate superintelligence. I guess I would frame the problem like this: > Let's say there is a very esoteric but important branch of abstract mathematics that only a few people claim to understand. Is there a way for us to determine which mathematicians are actually intelligent, and which are bluffing? How? This is a tricky one. To my dog, I am revered as a super-being of intelligence and capability. But if he watches me play grandmaster-level chess, or writing a paper on abstract mathematics -- it must look like insanity. In sci-fi, I rather like the image of super-intelligence from one of my favorite short-stories: "When the Yogurt Took Over" [1] > No one argues with the yogurt. No one tweaks its formulas. The rest of the time it rests there in its factory, thinking about whatever intelligent fermented milk thinks about. It just sits there in its vat -- and its actions seem largely incomprehensible to us -- as incomprehensible as me playing Magic: The Gathering is to my dog. It must look like lunacy. (given what I spend on the game, I'm not sure it's not) So if we're going to evaluate superintelligence, then I feel that -- for starters -- it must be on somewhat of a clear playing-field. We can clearly evaluate super-ability in Chess, in Go, and in Starcraft 2 because there are clearly defined rules. The only true test of whether one is superior to another will be because "it works". Until we can test abstract mathematics objectively, then I'm not sure we could ever judge. In so far as questions of particle physics and whatnot could actually be tested -- those feel like the sorts of areas where we might be able to evaluate superintelligence. But SPAG is much smaller than that. The hope that SPAG offers is that -- as long as the game rules leverage things like language and semantics -- then (assuming the model is able to generalize), then the increased mastery of language will transfer to other tasks. And the SPAG results seem to bear that out. [1] https://whatever.scalzi.com/2010/10/02/when-the-yogurt-took-over-a-short-story/ https://whatever.scalzi.com/2010/10/02/when-the-yogurt-took-...