23 ms·

I think this is both true and false in some senses. The whole point of the transformer and why it was so impactful is that it gave the model the ability to intr

I think this is both true and false in some senses. The whole point of the transformer and why it was so impactful is that it gave the model the ability to introspect on output in this way, and therefore achieve the sorts of results you've seen (that I've also seen), and in general a much more consistent output, and therefore one more likely to be factual, than previous types of generative model. However, the way in which the author is correct is that there is still functionally one stream, one "thought process", and the introspection is very limited and well defined based on how the model is constructed. If you ask a human a hard question they can take longer thinking about it, and that's not something an LLM can do at all. o1 is one attempt to add further streams of thought as a way to address that limitation, so that the LLM can continue to think and weigh up options through multiple rounds of generation.

anon291 2y ago
You have to think of the next token prediction as the immediate next 'thought'. We have trained LLMs to be forced to immediately say the right thing. This is akin to how real humans almost immediately will formulate an answer but then think before responding.. Except imagine if someone had a probe that looked into your mind and made you commit to the first thing you thought. Companies like anthropic literally train their model on the introspected thoughts of the earlier model (constitutional AI). The model is capable of 'thought' if the system doesn't force its hand
lxgr 2y ago
I certainly agree on "very limited". (Transformers are linear-time complexity per token, as far as I understand, so from computational theory principles, there are things these things trivially will not be able to answer, as they just "can't take longer thinking about it" as you point out.) But "very limited" isn't "not at all", and I think the distinction matters, since the former can often be scaled more easily than the latter. And at the meta-level, one possible output token can always be "let me ramble on this a bit more before I provide my final answer". In any case, I think "they can't look back at their output", as presented in the article, is just the wrong way to look at it; even "they can (currently) not take longer time to look at their output" is also wrong, since they can just produce more output, which then in turn lets them "think more".

anon291 2y ago

You have to think of the next token prediction as the immediate next 'thought'. We have trained LLMs to be forced to immediately say the right thing. This is akin to how real humans almost immediately will formulate an answer but then think before responding.. Except imagine if someone had a probe that looked into your mind and made you commit to the first thing you thought. Companies like anthropic literally train their model on the introspected thoughts of the earlier model (constitutional AI). The model is capable of 'thought' if the system doesn't force its hand

lxgr 2y ago

I certainly agree on "very limited". (Transformers are linear-time complexity per token, as far as I understand, so from computational theory principles, there are things these things trivially will not be able to answer, as they just "can't take longer thinking about it" as you point out.) But "very limited" isn't "not at all", and I think the distinction matters, since the former can often be scaled more easily than the latter. And at the meta-level, one possible output token can always be "let me ramble on this a bit more before I provide my final answer". In any case, I think "they can't look back at their output", as presented in the article, is just the wrong way to look at it; even "they can (currently) not take longer time to look at their output" is also wrong, since they can just produce more output, which then in turn lets them "think more".