4 ms·

> We’re also launching a feature to return the log probabilities for the most likely output tokens generated by GPT-4 Turbo and GPT-3.5 Turbo in the next few we

by gwern 3y ago

> We’re also launching a feature to return the log probabilities for the most likely output tokens generated by GPT-4 Turbo and GPT-3.5 Turbo in the next few weeks, which will be useful for building features such as autocomplete in a search experience. This is very surprising to me. Are they not worried about people not just training on GPT-4 outputs to steal the model capabilities, but doing full blown logit knowledge-distillation? (Which is the reason everyone assumed that they disabled logit access in the first place.)

leobg 3y ago
How many GBs worth of logits would you need to reverse engineer their model? Also, if it’s a conglomerate of models that they’re using, you’d end up in a blind alley.
- gwern 3y ago
  Considering how well simply reusing GPT-3.5/4 outputs has worked to juice rival model performance, at least in relatively narrow benchmarking, I dunno how many GBs it'd take, but probably not that many, and it's a straightforward easy way to turn money into performance at a much lower cost than buying a few thousand more H100s.
  - leobg 3y ago
    OpenAI does not strike me as a company that would be naive about this. Didn’t they just recently manipulate the outputs of an endpoint when they realized people were misusing it? (“CatGPT”) The most sinister interpretation is that the logits are a red herring. People who are tied up in stealing them aren’t free to do actual rival work.
    - joegibbs 3y ago
      What happened with CatGPT?
      - mike_hearn 3y ago
        Someone was abusing the ChatGPT backend to avoid paying for the API calls. When OpenAI's abuse team noticed this they modified the system prompt used for those requests to tell the AI to pretend it was a cat, and then monitored the Discord channel of the abusers for the lulz.
        msp26 3y ago
        >When OpenAI's abuse team noticed this they modified the system prompt used for those requests to tell the AI to pretend it was a cat, and then monitored the Discord channel of the abusers for the lulz. Please I need a source for this it sounds hilarious.
    - gwern 3y ago
      Well, not recently. That was from at the ChatGPT launch when they were still using a quick hack that made the client too-privileged, so more or less a year ago. However, I definitely am wondering if they have poisoned the logits somehow. As long as the logits rank the tokens in order (are monotonic), preserving their utility for the suggested applications like ranking autocompletions, you presumably could screw with the magnitudes arbitrarily.
danielmarkbruce 3y ago
I thought the same thing.... My guess is they did a lot of analysis and decided it would be safe enough to do? "most likely" might be literally a handful and cover little of the entire distribution % wise?
nwoli 3y ago
I guess the EO takes care of that in their eyes (outlawing open models). They’re probably right too
- dragonwriter 3y ago
  > I guess the EO takes care of that in their eyes (outlawing open models). The EO doesn't do anything even approximately like outlawing open models.
  - nwoli 3y ago
    It basically bans models that can be finetuned to act as a biological weapon which is basically every model, llama, mistral etc
    - dragonwriter 3y ago
      > t basically bans models that can be finetuned to act as a biological weapon which is basically every model, llama, mistral etc First, no, it doesn't. Second, no model can be finetuned to act as a biological weapon. Third, if it did ban “basically every model” on the basis “can be finetuned to act as a biological weapon”, that would be very different than banning open models, and would be bad for OpenAI. The order directs the development of reporting requirements for thise developing models with certain capabilities within 3 months, and the development within government of risk mitigation strategies for certain risks within 4, 6, or 9 months, depending on the specific area of risk. It doesn't ban or “basically ban” any models. (It is possible, but far from certain, that on or more of the plans it calls on different agencies to develop might do that, but those would, in addition to the policy guidance in the EO, also need a statutory authority that provides power for the executive branch to issue a ban.)

by gwern 3y ago

leobg 3y ago

How many GBs worth of logits would you need to reverse engineer their model? Also, if it’s a conglomerate of models that they’re using, you’d end up in a blind alley.

gwern 3y ago
Considering how well simply reusing GPT-3.5/4 outputs has worked to juice rival model performance, at least in relatively narrow benchmarking, I dunno how many GBs it'd take, but probably not that many, and it's a straightforward easy way to turn money into performance at a much lower cost than buying a few thousand more H100s.
- leobg 3y ago
  OpenAI does not strike me as a company that would be naive about this. Didn’t they just recently manipulate the outputs of an endpoint when they realized people were misusing it? (“CatGPT”) The most sinister interpretation is that the logits are a red herring. People who are tied up in stealing them aren’t free to do actual rival work.
  - joegibbs 3y ago
    What happened with CatGPT?
    - mike_hearn 3y ago
      Someone was abusing the ChatGPT backend to avoid paying for the API calls. When OpenAI's abuse team noticed this they modified the system prompt used for those requests to tell the AI to pretend it was a cat, and then monitored the Discord channel of the abusers for the lulz.
      - msp26 3y ago
        >When OpenAI's abuse team noticed this they modified the system prompt used for those requests to tell the AI to pretend it was a cat, and then monitored the Discord channel of the abusers for the lulz. Please I need a source for this it sounds hilarious.
  - gwern 3y ago
    Well, not recently. That was from at the ChatGPT launch when they were still using a quick hack that made the client too-privileged, so more or less a year ago. However, I definitely am wondering if they have poisoned the logits somehow. As long as the logits rank the tokens in order (are monotonic), preserving their utility for the suggested applications like ranking autocompletions, you presumably could screw with the magnitudes arbitrarily.

danielmarkbruce 3y ago

I thought the same thing.... My guess is they did a lot of analysis and decided it would be safe enough to do? "most likely" might be literally a handful and cover little of the entire distribution % wise?

nwoli 3y ago

I guess the EO takes care of that in their eyes (outlawing open models). They’re probably right too

dragonwriter 3y ago
> I guess the EO takes care of that in their eyes (outlawing open models). The EO doesn't do anything even approximately like outlawing open models.
- nwoli 3y ago
  It basically bans models that can be finetuned to act as a biological weapon which is basically every model, llama, mistral etc
  - dragonwriter 3y ago
    > t basically bans models that can be finetuned to act as a biological weapon which is basically every model, llama, mistral etc First, no, it doesn't. Second, no model can be finetuned to act as a biological weapon. Third, if it did ban “basically every model” on the basis “can be finetuned to act as a biological weapon”, that would be very different than banning open models, and would be bad for OpenAI. The order directs the development of reporting requirements for thise developing models with certain capabilities within 3 months, and the development within government of risk mitigation strategies for certain risks within 4, 6, or 9 months, depending on the specific area of risk. It doesn't ban or “basically ban” any models. (It is possible, but far from certain, that on or more of the plans it calls on different agencies to develop might do that, but those would, in addition to the policy guidance in the EO, also need a statutory authority that provides power for the executive branch to issue a ban.)