4 ms·
How would an LLM “know” when it isn’t sure? Their baseline for truth is competent text, they don’t have a baseline for truth based on observed reality. That’s w
by dkdbejwi383 1y ago
How would an LLM “know” when it isn’t sure? Their baseline for truth is competent text, they don’t have a baseline for truth based on observed reality. That’s why they can be “tricked” into things like “Mr Bean is the president of the USA”
- ben_w 1y agoThe answer is the same as how the messy bag of chemistry that is the human brain "knows" when it isn't sure: Badly, and with great difficulty, so while it can just about be done, even then only kinda.
- foldr 1y agoWe really don’t understand the human brain well enough to have confidence that the mechanisms that cause people to respond with “I don’t know” are at all similar to the mechanisms which cause LLMs to give such responses. And there are quite a few prima facie reasons to think that they wouldn’t be the same.
- Sharlin 1y agoThe mechanics don't have to be similar, only analogous, in the morphology sense.
- foldr 1y ago'Analogous in the morphology sense' is actually a more specific concept than 'similar'. But either way, we still don't know if they're analogous, or similar, or whatever term you prefer. Anyone who actually understands both LLMs and the human brain well enough to make confident claims that they basically work the same really ought to put in the effort to write up a paper and get a Nobel prize or two.
- Sharlin 1y agoAnalogous in the morphology sense means having come up with an entirely distinct solution to a common problem. Insect and bird wings have little to do with each other except that both flap to create lift. It explicitly does not imply the solutions are similar in mechanism, although that can be, and often is, a result of convergent evolution, of course. In particular, generally speaking (not claiming that LLMs a road to AGI, which is something I doubt) it's generally not a well-defensible philosophical position that the vertebrate brain (and remember that mammalian, bird and cephalopod brains are very different) is uniquely suited to produce what we call "intelligence". > Anyone who actually understands both LLMs and the human brain well enough to make confident claims that they basically work the same This is a strawman and not my position.
- foldr 1y agoIt was a characterization of the position of the post I was originally responding to, not your position. I don’t think anyone in this discussion has claimed that brains are uniquely suited to producing intelligence. The point was just that we have no idea if there is any interesting correspondence between how LLMs work and how brains work, beyond superficial and obvious analogies.
- ben_w 1y agoFWIW, I'm describing failure modes of a human, not mechanisms. I also think "would" in the comment I'm replying to is closer to "could" than to "does".
- foldr 1y agoCould you expand on that? What failure modes are we talking about exactly?
- ben_w 1y agoDunning Kruger, and also everyone who thinks they know better than domain experts. Humans who are wrong are often completely oblivious to being wrong.
- hglover945 1y ago[dead]
- foldr 1y agoOk, but what does that have to do with LLMS?
- ben_w 1y agoI'm saying both human brains and LLMs are bad at knowing when they're unsure — it may not even be possible to make an LLM that's superhuman in this regard. Most random humans have nearly constant confidence in their answers regardless of how much they know about a topic. -- Hmm. Stream of consciousness thoughts here: Problem is LLMs are very broad "System 1" thinking, no "System 2". Other kinds of model that combine LLM with a more logical module? Perhaps that could work. And if so then I guess you could use the logical module to create tokens for an LLM to learn from? Eventually, but may be inefficient. To work, logic module needs to know what LLM gets wrong, to create tokens for LLM to learn from; if you can do that, but fine tuning is slow and expensive and still errs, so why not keep the logic separate and just have it point out mistakes and have LLM regenerate answers a few times and if that doesn't work then say "IDK"?
- JustFinishedBSG 1y agoIt would "know" the same way it "knows" anything else: The probability of the sequence "I don't know" would be higher than the probability of any other sequence.
- Sharlin 1y agoExactly. It's easy to imagine a component in the net that the model is steered towards when nothing else has a high enough activation.
- saberience 1y agoHumans can just as easily be tricked. Something like 25% of the American Electorate believed Obama was the antichrist. So saying LLMs have no "baseline for truth" doesn't really mean much one way of the other, they are much smart and accurate than 99% of humans.