3 ms·

This is an interesting premise that should pursuit further testing on base models rather than instruction-enhanced ones. The assumption that LLMs with all its k

This is an interesting premise that should pursuit further testing on base models rather than instruction-enhanced ones. The assumption that LLMs with all its knowledge is able to ballpark a potentially quantifiable guess is not illogical.

famouswaffles 3y ago
That marks an interesting question though. Why is base GPT-4 superbly calibrated when humans are not ?
- MacsHeadroom 3y ago
  Because the human reward function is to reproduce human genetic code while an LLM's reward function is to reproduce the training data.
  - LastTrain 3y ago
    Isn’t there an implicit survival reward lurking there. The ones that don’t reproduce the training data to our satisfaction die off.
    - nyc_data_geek1 3y ago
      They're not "trying" to survive or striving for any "reward". They're a function with no independent awareness.
      - chii 3y ago
        Just like how bacteria has no independent "awareness" (as far as we know). And yet, they are partaking in darwinian selection. It's just in the world of LLM, it's not nature but artificial, and that "reproduction" is not simple and random like it would be in nature, but via careful selection of humans.
        mannykannot 3y ago
        Like domestication, then.
  - dongobread 3y ago
    There's a bunch of research on how optimism bias (which I imagine is very closely linked to overconfidence) can be evolutionarily driven, e.g. https://www.sciencedirect.com/science/article/pii/S0960982211011912 https://www.sciencedirect.com/science/article/pii/S096098221...
- gwern 3y ago
  'Wisdom of the crowds'. Any individual human is uncalibrated (although this is fairly easy to train away) and makes errors, but a LLM is predicting the distribution of all human responses as reflected in the corpus. So it reports a reasonable distribution.
- kromem 3y ago
  Because while the model at baseline is a regression towards the mean, it is easily biased based on additional context - one of the big eye openers in research over the past 18 months has been the impact and success of in context learning alone. So with an LLM you have a very broad basis that can be biased into a niche application in a single unit whereas with humans you have a fairly static set of capabilities that are widely and normally distributed across individual units. You can be lucky in finding the right unit for the task with humans that are extremely good at extending a given context. But you need to have been lucky with matching the unit's performance with the right context. A different unit will suck. But LLMs given enough coverage in the training set and enough additional context to bias towards that subset of the training set can be consistently good (and will likely continue to get better).
refulgentis 3y ago
There's not really any reason to think this other than shibboleths, it's become a thought-deadening cliche. I've played with true base models at big tech over a period of 6+ months, it's a much much more frustrating experience, not an "unlocked" or "honest" version. Setting temperature to 1.2 is a reasonable miniature taste of the vibe of it. It would be interesting to see it done using logit probabilities, I.e. instead of asking for p 0 to 1, instead, ask it true or false, set logit bias such that those are the only two tokens that could be output, and use the logit probability emitted by the API.
- gwern 3y ago
  > There's not really any reason to think this other than shibboleths, it's become a thought-deadening cliche. The official OpenAI GPT-4 paper specifically reporting that the RLHF model is badly uncalibrated, while the GPT-4 base mode is pretty well calibrated, is a good reason to think that, and is not a 'thought-deadening cliche shibboleth': https://arxiv.org/pdf/2303.08774.pdf#page=12 https://arxiv.org/pdf/2303.08774.pdf#page=12 "The post-training hurts calibration significantly."
  - refulgentis 3y ago
    Fascinating - thank you!!
bertil 3y ago
I would ask it to justify its decisions in detail and ask an expert to judge the reasoning. I'm still routinely disappointed by LLMs failing basic reasoning tasks. Most LLMs fail easy tests like "A is faster than B, B is faster than C, is C faster than A?" and questions about a ball in an upside-down cup. Better models get those but fail at other “common sense” tasks. One example from the very end of the OpenAI keynote, the demo when they granted credits first to five people and then to everyone: did the AI know not to credit the five lucky recipients twice?

famouswaffles 3y ago

That marks an interesting question though. Why is base GPT-4 superbly calibrated when humans are not ?

MacsHeadroom 3y ago
Because the human reward function is to reproduce human genetic code while an LLM's reward function is to reproduce the training data.
- LastTrain 3y ago
  Isn’t there an implicit survival reward lurking there. The ones that don’t reproduce the training data to our satisfaction die off.
  - nyc_data_geek1 3y ago
    They're not "trying" to survive or striving for any "reward". They're a function with no independent awareness.
    - chii 3y ago
      Just like how bacteria has no independent "awareness" (as far as we know). And yet, they are partaking in darwinian selection. It's just in the world of LLM, it's not nature but artificial, and that "reproduction" is not simple and random like it would be in nature, but via careful selection of humans.
      - mannykannot 3y ago
        Like domestication, then.
- dongobread 3y ago
  There's a bunch of research on how optimism bias (which I imagine is very closely linked to overconfidence) can be evolutionarily driven, e.g. https://www.sciencedirect.com/science/article/pii/S0960982211011912 https://www.sciencedirect.com/science/article/pii/S096098221...
gwern 3y ago
'Wisdom of the crowds'. Any individual human is uncalibrated (although this is fairly easy to train away) and makes errors, but a LLM is predicting the distribution of all human responses as reflected in the corpus. So it reports a reasonable distribution.
kromem 3y ago
Because while the model at baseline is a regression towards the mean, it is easily biased based on additional context - one of the big eye openers in research over the past 18 months has been the impact and success of in context learning alone. So with an LLM you have a very broad basis that can be biased into a niche application in a single unit whereas with humans you have a fairly static set of capabilities that are widely and normally distributed across individual units. You can be lucky in finding the right unit for the task with humans that are extremely good at extending a given context. But you need to have been lucky with matching the unit's performance with the right context. A different unit will suck. But LLMs given enough coverage in the training set and enough additional context to bias towards that subset of the training set can be consistently good (and will likely continue to get better).

refulgentis 3y ago

There's not really any reason to think this other than shibboleths, it's become a thought-deadening cliche. I've played with true base models at big tech over a period of 6+ months, it's a much much more frustrating experience, not an "unlocked" or "honest" version. Setting temperature to 1.2 is a reasonable miniature taste of the vibe of it. It would be interesting to see it done using logit probabilities, I.e. instead of asking for p 0 to 1, instead, ask it true or false, set logit bias such that those are the only two tokens that could be output, and use the logit probability emitted by the API.

gwern 3y ago
> There's not really any reason to think this other than shibboleths, it's become a thought-deadening cliche. The official OpenAI GPT-4 paper specifically reporting that the RLHF model is badly uncalibrated, while the GPT-4 base mode is pretty well calibrated, is a good reason to think that, and is not a 'thought-deadening cliche shibboleth': https://arxiv.org/pdf/2303.08774.pdf#page=12 https://arxiv.org/pdf/2303.08774.pdf#page=12 "The post-training hurts calibration significantly."
- refulgentis 3y ago
  Fascinating - thank you!!

bertil 3y ago

I would ask it to justify its decisions in detail and ask an expert to judge the reasoning. I'm still routinely disappointed by LLMs failing basic reasoning tasks. Most LLMs fail easy tests like "A is faster than B, B is faster than C, is C faster than A?" and questions about a ball in an upside-down cup. Better models get those but fail at other “common sense” tasks. One example from the very end of the OpenAI keynote, the demo when they granted credits first to five people and then to everyone: did the AI know not to credit the five lucky recipients twice?