3 ms·

This is a really bizarre thing to do honestly It's plausible that there are potential avenues for improving language models through adversarial learning. GANs

by advael 2y ago

This is a really bizarre thing to do honestly It's plausible that there are potential avenues for improving language models through adversarial learning. GANs and Actor-Critic models have done a good job in narrow-domain generative applications and task learning, and I can make a strong theoretical argument that you can do something that looks like priority learning via adversarial equilibria But why in the world are you trying to present this as a human-in-the-loop system? This makes no sense to me. You take an error-prone generative language model and then present another instance of an error-prone generative language model to "critique" it for the benefit of... a human observer? The very best case here is that this wastes a bunch of heat and time for what can only be a pretty nebulous potential gain to the human's understanding Is this some weird gambit to get people to trust these models more? Is it OpenAI losing the plot completely because they're unwilling to go back to open-sourcing their models but addicted to the publicity of releasing public-facing interfaces to them? This doesn't make sense to me as a research angle or as a product I can really see the Microsoft influence here

kenjackson 2y ago
It's for their RLHF pipeline to improve labeling. Honestly, this seems super reasonable to me. I don't get why you think this is such a bad idea for this purpose...
- advael 2y ago
  RLHF to me seems more as a PR play than anything else, but inasmuch as it does anything useful, adding a second LLM to influence the human that's influencing the LLM doesn't solve any of the fundamental problems of either system. If anything it muddies the waters more, because we have already seen that humans are probably too credulous of the information presented to them by these models. If you want adversarial learning, there are far more efficient ways to do it. If you want human auditing, the best case here is that the second LLM doesn't influence the human's decisions at all (because any influence reduces the degree to which this is independent feedback)
  - vhiremath4 2y ago
    This is kind of what I was thinking. I don’t get it. It seems like CriticGPT was maybe trained using RM/RL with PPO as well? So there’s gonna be mistakes with what CriticGPT pushes back on which may make the labeler doubt themselves?
  - kenjackson 2y ago
    This is not adversarial learning. It's really about augmenting the ability of humans to determine if a snippet of code is correct and write proper critiques of incorrect code. Any system that helps you more accurately label data with good critiques should help the model. I'm not sure how you come to your conclusion. Do you have some data to indicate that even with improved accuracy that some LLM bias would lead to a worse trained model? I haven't seen that data or assertion elsewhere, but that's the only thing I can gather you might be referring.
    - advael 2y ago
      Well, first of all, the stated purpose of RLHF isn't to "improve model accuracy" in the first place (and what we mean by accuracy here is pretty fraught by itself, as this could mean at least three different things). They initially pitched it as a "safety" measure (and I think if it wasn't obvious immediately how nonsensical a claim that is, it should at least be apparent now that the company's shucked nearly the entire subset of its members that claimed to care about "AI safety" that this is not a priority) The idea of RLHF as a mechanism for tuning models based on the principle that humans might have some hard-to-capture insight that could steer them independent of the way they're normally trained is the very best steelman for its value I could come up with. This aim is directly subverted by trying to use another language model to influence the human rater, so from my perspective it really brings us back to square one on what the fuck RLHF is supposed to be doing Really, a lot of this comes down to what these models do versus how they are being advertised. A generative language model produces plausible prose that follows from the prompt it receives. From this, the claim that it should write working code is actually quite a bit stronger than the claim that it should write true facts, because plausibile autocompletion will learn to mimic syntactic constraints but actually has very little to do with whether something is true, or whatever proxy or heuristic we may apply in place of "true" when assessing information (supported by evidence, perhaps. Logically sound, perhaps. The distinction between "plausible" and "true" is in many ways the whole point of every human epistemology). Like if you ask something trained on all human writing whether the Axis or the Allies won WWII, the answer will depend on whether you phrased the question in a way that sounds like Phillip K Dick would write it. This isn't even incorrect behavior by the standards of the model, but people want to use these things like some kind of oracle or to replace google search or whatever, which is a misconception about what the thing does, and one that's very profitable for the people selling it

by advael 2y ago

kenjackson 2y ago

It's for their RLHF pipeline to improve labeling. Honestly, this seems super reasonable to me. I don't get why you think this is such a bad idea for this purpose...

advael 2y ago
RLHF to me seems more as a PR play than anything else, but inasmuch as it does anything useful, adding a second LLM to influence the human that's influencing the LLM doesn't solve any of the fundamental problems of either system. If anything it muddies the waters more, because we have already seen that humans are probably too credulous of the information presented to them by these models. If you want adversarial learning, there are far more efficient ways to do it. If you want human auditing, the best case here is that the second LLM doesn't influence the human's decisions at all (because any influence reduces the degree to which this is independent feedback)
- vhiremath4 2y ago
  This is kind of what I was thinking. I don’t get it. It seems like CriticGPT was maybe trained using RM/RL with PPO as well? So there’s gonna be mistakes with what CriticGPT pushes back on which may make the labeler doubt themselves?
- kenjackson 2y ago
  This is not adversarial learning. It's really about augmenting the ability of humans to determine if a snippet of code is correct and write proper critiques of incorrect code. Any system that helps you more accurately label data with good critiques should help the model. I'm not sure how you come to your conclusion. Do you have some data to indicate that even with improved accuracy that some LLM bias would lead to a worse trained model? I haven't seen that data or assertion elsewhere, but that's the only thing I can gather you might be referring.
  - advael 2y ago
    Well, first of all, the stated purpose of RLHF isn't to "improve model accuracy" in the first place (and what we mean by accuracy here is pretty fraught by itself, as this could mean at least three different things). They initially pitched it as a "safety" measure (and I think if it wasn't obvious immediately how nonsensical a claim that is, it should at least be apparent now that the company's shucked nearly the entire subset of its members that claimed to care about "AI safety" that this is not a priority) The idea of RLHF as a mechanism for tuning models based on the principle that humans might have some hard-to-capture insight that could steer them independent of the way they're normally trained is the very best steelman for its value I could come up with. This aim is directly subverted by trying to use another language model to influence the human rater, so from my perspective it really brings us back to square one on what the fuck RLHF is supposed to be doing Really, a lot of this comes down to what these models do versus how they are being advertised. A generative language model produces plausible prose that follows from the prompt it receives. From this, the claim that it should write working code is actually quite a bit stronger than the claim that it should write true facts, because plausibile autocompletion will learn to mimic syntactic constraints but actually has very little to do with whether something is true, or whatever proxy or heuristic we may apply in place of "true" when assessing information (supported by evidence, perhaps. Logically sound, perhaps. The distinction between "plausible" and "true" is in many ways the whole point of every human epistemology). Like if you ask something trained on all human writing whether the Axis or the Allies won WWII, the answer will depend on whether you phrased the question in a way that sounds like Phillip K Dick would write it. This isn't even incorrect behavior by the standards of the model, but people want to use these things like some kind of oracle or to replace google search or whatever, which is a misconception about what the thing does, and one that's very profitable for the people selling it