4 ms·

> Humans would say "Left" 100% of the time in a zero-shot scenario as well. How can you know what all humans would do? If the humans interpreted the task corr

> Humans would say "Left" 100% of the time in a zero-shot scenario as well. How can you know what all humans would do? If the humans interpreted the task correctly, that is, if they understood they will only be asked once, but in a hypothetical repeated experiment the result should still be 80/20, they would certainly not always say "left".

bena 2y ago
Because it's a stupid prompt. Especially for humans. Because you're really asking what they think the first response would be. That's left. If I knew a machine would pick left 80% of the time, I would bet left 100% of the time. And I'd be right about 80% of the time, which isn't perfect, but is profitable.
ben_w 2y ago
A human brain can't be perfectly reset, the way an AI can. I don't know if our decision making processes are deterministic or quantum-random. If the former, then if you could reset a human mind and ask the same question, you would necessarily always get the same answer, whatever that happened to be.
- hwillis 2y ago
  The LLM isn't being perfectly reset. It chooses words randomly; internally it should be slightly different every time. That's the whole point of temperature.
  - tempusalaria 2y ago
    Temperature has nothing to do with internals. Temperature is purely to do with how the logits outputted by the network are transformed into probabilities, which is completely deterministic and not learned. In fact, temperature makes it impossible for LLMs to simulate this kind of probability. As a calibrated 80-20 split at a certain low temperature would be a different split with some other temperature.
  - gudd0g 2y ago
    [dead]
itsgrimetime 2y ago
assuming the humans don’t know what the other responses were, I can’t imagine it actually coming out 80/20
- sqeaky 2y ago
  When polls like these are run the numbers don't always wind up tilted in the favor of the bigger number. I wish I could provide a specific source but I've been listening to the 538 podcast for years and I know they've covered exactly this topic. Your inability to believe a thing doesn't prevent it from being true. I would grab a D20 and on a 16 or less I would say left otherwise I would say right. Some people would pick right just because they can. I imagine most people would pick left because it's the 80%. I imagine plenty of people would double and triple guess and waffle then say something. Few people, even the dumbest among us, are easily modelable deterministic automata.
- bmacho 2y ago
  Because some humans don't even understand the question. Others do, but they don't know how to solve it. But other humans can do this task. If you do it with a bunch of mathematicians in a room, you will get your 20-80 distribution.

bena 2y ago

Because it's a stupid prompt. Especially for humans. Because you're really asking what they think the first response would be. That's left. If I knew a machine would pick left 80% of the time, I would bet left 100% of the time. And I'd be right about 80% of the time, which isn't perfect, but is profitable.

ben_w 2y ago

A human brain can't be perfectly reset, the way an AI can. I don't know if our decision making processes are deterministic or quantum-random. If the former, then if you could reset a human mind and ask the same question, you would necessarily always get the same answer, whatever that happened to be.

hwillis 2y ago
The LLM isn't being perfectly reset. It chooses words randomly; internally it should be slightly different every time. That's the whole point of temperature.
- tempusalaria 2y ago
  Temperature has nothing to do with internals. Temperature is purely to do with how the logits outputted by the network are transformed into probabilities, which is completely deterministic and not learned. In fact, temperature makes it impossible for LLMs to simulate this kind of probability. As a calibrated 80-20 split at a certain low temperature would be a different split with some other temperature.
- gudd0g 2y ago
  [dead]

itsgrimetime 2y ago

assuming the humans don’t know what the other responses were, I can’t imagine it actually coming out 80/20

sqeaky 2y ago
When polls like these are run the numbers don't always wind up tilted in the favor of the bigger number. I wish I could provide a specific source but I've been listening to the 538 podcast for years and I know they've covered exactly this topic. Your inability to believe a thing doesn't prevent it from being true. I would grab a D20 and on a 16 or less I would say left otherwise I would say right. Some people would pick right just because they can. I imagine most people would pick left because it's the 80%. I imagine plenty of people would double and triple guess and waffle then say something. Few people, even the dumbest among us, are easily modelable deterministic automata.
bmacho 2y ago
Because some humans don't even understand the question. Others do, but they don't know how to solve it. But other humans can do this task. If you do it with a bunch of mathematicians in a room, you will get your 20-80 distribution.