3 ms·

That marks an interesting question though. Why is base GPT-4 superbly calibrated when humans are not ?

MacsHeadroom 3y ago
Because the human reward function is to reproduce human genetic code while an LLM's reward function is to reproduce the training data.
- LastTrain 3y ago
  Isn’t there an implicit survival reward lurking there. The ones that don’t reproduce the training data to our satisfaction die off.
  - nyc_data_geek1 3y ago
    They're not "trying" to survive or striving for any "reward". They're a function with no independent awareness.
    - chii 3y ago
      Just like how bacteria has no independent "awareness" (as far as we know). And yet, they are partaking in darwinian selection. It's just in the world of LLM, it's not nature but artificial, and that "reproduction" is not simple and random like it would be in nature, but via careful selection of humans.
      - mannykannot 3y ago
        Like domestication, then.
- dongobread 3y ago
  There's a bunch of research on how optimism bias (which I imagine is very closely linked to overconfidence) can be evolutionarily driven, e.g. https://www.sciencedirect.com/science/article/pii/S0960982211011912 https://www.sciencedirect.com/science/article/pii/S096098221...
gwern 3y ago
'Wisdom of the crowds'. Any individual human is uncalibrated (although this is fairly easy to train away) and makes errors, but a LLM is predicting the distribution of all human responses as reflected in the corpus. So it reports a reasonable distribution.
kromem 3y ago
Because while the model at baseline is a regression towards the mean, it is easily biased based on additional context - one of the big eye openers in research over the past 18 months has been the impact and success of in context learning alone. So with an LLM you have a very broad basis that can be biased into a niche application in a single unit whereas with humans you have a fairly static set of capabilities that are widely and normally distributed across individual units. You can be lucky in finding the right unit for the task with humans that are extremely good at extending a given context. But you need to have been lucky with matching the unit's performance with the right context. A different unit will suck. But LLMs given enough coverage in the training set and enough additional context to bias towards that subset of the training set can be consistently good (and will likely continue to get better).

Because the human reward function is to reproduce human genetic code while an LLM's reward function is to reproduce the training data.

LastTrain 3y ago
Isn’t there an implicit survival reward lurking there. The ones that don’t reproduce the training data to our satisfaction die off.
- nyc_data_geek1 3y ago
  They're not "trying" to survive or striving for any "reward". They're a function with no independent awareness.
  - chii 3y ago
    Just like how bacteria has no independent "awareness" (as far as we know). And yet, they are partaking in darwinian selection. It's just in the world of LLM, it's not nature but artificial, and that "reproduction" is not simple and random like it would be in nature, but via careful selection of humans.
    - mannykannot 3y ago
      Like domestication, then.
dongobread 3y ago
There's a bunch of research on how optimism bias (which I imagine is very closely linked to overconfidence) can be evolutionarily driven, e.g. https://www.sciencedirect.com/science/article/pii/S0960982211011912 https://www.sciencedirect.com/science/article/pii/S096098221...

gwern 3y ago

'Wisdom of the crowds'. Any individual human is uncalibrated (although this is fairly easy to train away) and makes errors, but a LLM is predicting the distribution of all human responses as reflected in the corpus. So it reports a reasonable distribution.

kromem 3y ago

Because while the model at baseline is a regression towards the mean, it is easily biased based on additional context - one of the big eye openers in research over the past 18 months has been the impact and success of in context learning alone. So with an LLM you have a very broad basis that can be biased into a niche application in a single unit whereas with humans you have a fairly static set of capabilities that are widely and normally distributed across individual units. You can be lucky in finding the right unit for the task with humans that are extremely good at extending a given context. But you need to have been lucky with matching the unit's performance with the right context. A different unit will suck. But LLMs given enough coverage in the training set and enough additional context to bias towards that subset of the training set can be consistently good (and will likely continue to get better).