5 ms·
why shouldn’t i assume that the “thinking” is just the usual LLM regurgitation of “how would a human coming up with a joke explain their reasoning?” or somethin
by laweijfmvo 2y ago
why shouldn’t i assume that the “thinking” is just the usual LLM regurgitation of “how would a human coming up with a joke explain their reasoning?” or something like that, and zero “thinking”?
- brookst 2y agoWhat’s the difference?
- bitwalker 2y agoAt least to me, the difference is that one is ostensibly an explanation of how the AI arrived at the joke, the other is a post-hoc explanation of the joke.
- gpm 2y agoYou can be pretty sure the AI isn't doing a post-hoc explanation because the only writable memory it has access to is the tokens it has already output - i.e. the explanation of the joke. Everything else is reset between every token. As long as it comes up with different jokes different times you ask it (assuming randomness in sampling) - how could it.
- baq 2y agoThe problem is it can’t remember what it hasn’t written but the end result still makes sense, so there has to be some goal after parsing the initial context that the tokens are emitted towards to. This means there’s nothing stopping it from producing an explanation, it might be in there from the very start.
- svachalek 2y agoThere's no goal. The tokens are computed one at a time from all the previous tokens.
- baq 2y agoOne is orthogonal to the other.
- gpm 2y agoNot one goal though, but a set of goals. Otherwise the random sampling of tokens wouldn't result in it outputting meaningfully different jokes. I also think it's safe to assume that the set of goals isn't fully resolved, but it's harder to "prove" that.
- barrkel 2y agoYou shouldn't assume that because you shouldn't assume the converse either. The LLM will use the chain of thought, as it's being built out like a garden path, to explore different completion possibilities. It's not necessarily logically related what it completes afterwards, but it's definitely influenced. The search may not be valuable as an artifact in itself. It's likely to be logically unsound especially in parts. But the end result may be useful.
- wrsh07 2y agoGiven how it is trained specifically (they didn't encourage it to think, they allowed it to) there was a lot of emergent behavior as it trained. Sort of like chess engines rediscovering classic (named) chess openings. See section 2.2.3 for the training template (it's a single paragraph I can't reproduce here because I'm on my phone) Example emergent behavior (section 2.2.4 page 8): the model learns to solve more complex problems by spending more time reasoning. It also naturally develops reflection (what have I tried?) and exploration strategies. Fundamentally, you should think of this as a nn that learned to solve real problems by reasoning about them in written language. (My favorite part: it defaulted to reasoning in multiple languages. They constrained it to only reason in a single language and this negatively impacted performance! But the hypothesis is that it improves interpretability)
- mordae 2y agoEver read philosophy? An acquaintance can and will readily mix and match 3 languages to obtain more precision.