4 ms·

Very interesting! The one thing I don't understand is how the author made the jump from "we lost the confidence signal in the move to 4.1-mini" and "this is bec

Very interesting! The one thing I don't understand is how the author made the jump from "we lost the confidence signal in the move to 4.1-mini" and "this is because of the alignment/steerability improvements." Previous OpenAI models were instruct-tuned or otherwise aligned, and the author even mentions that model distillation might be destroying the entropy signal. How did they pinpoint alignment as the cause?

mlin4589 1y ago
Good question! We do know from OpenAI's system card from GPT-4 that the post-trained RLHF model is significantly less calibrated compared to the pre-trained model, so it's a matter of speculation that something similar is occurring. However, it's more of a hunch more than anything. I would be curious if it's possible to reproduce this behavior, or the impact of distillation on calibration. Disclaimer: I wrote this blog post.
- itchyjunk 1y ago
  Could you please elaborate what less or more calibrated means here? Thanks!
  - Scene_Cast2 1y ago
    For binary labels: you take a slice of labeled data. The mean of the ML model prediction on this data is different from the mean of the label. In practice, often a synonym for "loss is worse / could be better". Not sure if that's what the GP meant, I only worked with binary labels stuff.
  - mlin4589 1y ago
    Calibration (in a binary context) basically means that the confidence of a model/score matches the probability that a particular label is positive or not. For instance, a calibrated classifier for a coin flip predictor should output 50-50. A poorly calibrated classifier would output higher confidence for heads/tails.
- Workaccount2 1y ago
  Wouldn't it be something if AI parlance crept into common parlance...
  - bluefirebrand 1y ago
    Great Observation! It would probably erode trust between people interacting online. Many of us are here to discuss issues with real people, not AI agents. When real people start to mimic the conversation parlance and cadence of AI agents it becomes much more difficult to trust that you are interacting with a real person Personally I'm not interested in chatting with AI agents I'm not even really interested in chatting with real people filtered through AI agents. If you can be bothered to type out a prompt to your AI you can take the time to write your own thoughts I don't even want to read things edited (sanitized, really) by AI either The same way I don't want my living space to resemble a too-clean laboratory, I don't want my conversation space to resemble an HR meeting. I want to interact with the messy side of people too. Maybe not "unfiltered", but AI speak is much too filtered and too polished I chose every word in this post myself with no help from AI, then typed it with my thumbs, just like god intended
    - dinfinity 1y ago
      > Personally I'm not interested in chatting with AI agents Why, though? If the AI agent is making sense, then what does it matter? For certain types of conversations I've had more interesting conversations with AI than with a solid 90% of people I've ever interacted with. Really not that surprising given that most people have only an average grasp of most things and a poor grasp of very specific things.
      - bluefirebrand 1y ago
        > Why, though? If the AI agent is making sense, then what does it matter? The same reason I prefer having sex with humans and not blow up dolls If your only goal is to get off, then the blow up doll does the job. If all you care about is having an interesting conversation then I guess an LLM is fine I care about human connection. I have no interest in spending time interacting with machines instead of people

mlin4589 1y ago

Good question! We do know from OpenAI's system card from GPT-4 that the post-trained RLHF model is significantly less calibrated compared to the pre-trained model, so it's a matter of speculation that something similar is occurring. However, it's more of a hunch more than anything. I would be curious if it's possible to reproduce this behavior, or the impact of distillation on calibration. Disclaimer: I wrote this blog post.

itchyjunk 1y ago
Could you please elaborate what less or more calibrated means here? Thanks!
- Scene_Cast2 1y ago
  For binary labels: you take a slice of labeled data. The mean of the ML model prediction on this data is different from the mean of the label. In practice, often a synonym for "loss is worse / could be better". Not sure if that's what the GP meant, I only worked with binary labels stuff.
- mlin4589 1y ago
  Calibration (in a binary context) basically means that the confidence of a model/score matches the probability that a particular label is positive or not. For instance, a calibrated classifier for a coin flip predictor should output 50-50. A poorly calibrated classifier would output higher confidence for heads/tails.
Workaccount2 1y ago
Wouldn't it be something if AI parlance crept into common parlance...
- bluefirebrand 1y ago
  Great Observation! It would probably erode trust between people interacting online. Many of us are here to discuss issues with real people, not AI agents. When real people start to mimic the conversation parlance and cadence of AI agents it becomes much more difficult to trust that you are interacting with a real person Personally I'm not interested in chatting with AI agents I'm not even really interested in chatting with real people filtered through AI agents. If you can be bothered to type out a prompt to your AI you can take the time to write your own thoughts I don't even want to read things edited (sanitized, really) by AI either The same way I don't want my living space to resemble a too-clean laboratory, I don't want my conversation space to resemble an HR meeting. I want to interact with the messy side of people too. Maybe not "unfiltered", but AI speak is much too filtered and too polished I chose every word in this post myself with no help from AI, then typed it with my thumbs, just like god intended
  - dinfinity 1y ago
    > Personally I'm not interested in chatting with AI agents Why, though? If the AI agent is making sense, then what does it matter? For certain types of conversations I've had more interesting conversations with AI than with a solid 90% of people I've ever interacted with. Really not that surprising given that most people have only an average grasp of most things and a poor grasp of very specific things.
    - bluefirebrand 1y ago
      > Why, though? If the AI agent is making sense, then what does it matter? The same reason I prefer having sex with humans and not blow up dolls If your only goal is to get off, then the blow up doll does the job. If all you care about is having an interesting conversation then I guess an LLM is fine I care about human connection. I have no interest in spending time interacting with machines instead of people