3 ms·
> It seems unlikely that humans think in “tokens” so why do LLMs? I can think of one reason: scrutability. It’s going to be even harder to understand how a res
by otikik 2y ago
> It seems unlikely that humans think in “tokens” so why do LLMs?
I can think of one reason: scrutability. It’s going to be even harder to understand how a response gets produced if there isn’t even a text-based representation to help the human understand
- IshKebab 2y agoI think we're already way beyond the point where anyone really understands how a response is produced, even without this.
- nfw2 2y agothe token generation part isn't well understood, but the output "chain-of-thought" used to produce the final answer can be scrutinized for correctness with a traditional CoT model (although this would require model providers to not hide reasoning tokens)
- anon373839 2y agoIndeed. Even if an LLM tells you its “reasoning” process step by step, it’s not actually an exposition of the model’s internal decision process. It’s just more text that, when generated, improves the chances of a good final output.
- deleted 2y ago[deleted]
- pigpop 2y agoyou can save the hidden states and convert them into a more interpretable format. it's still recorded and you could make modifications at different steps to see how that would change the conclusion.