5 ms·

If chain of thought acts as a scratch buffer by providing the model more temporary "layers" to process the text, I wonder if making this buffer a separate conte

by pona-a 2y ago

If chain of thought acts as a scratch buffer by providing the model more temporary "layers" to process the text, I wonder if making this buffer a separate context with its own separate FNN and attention would make sense; in essence, there's a macroprocess of "reasoning" that takes unbounded time to complete, and then there's a microprocess of describing this incomprehensible stream of embedding vectors in natural language, in a way returning to the encoder/decoder architecture but where both are autoregressive. Maybe this would give us a denser representation of said "thought", not constrained by imitating human text.

bluechair 2y ago
I had this exact same thought yesterday. I’d go so far as to add one more layer to monitor this one and stop adding layers. My thinking is that this meta awareness is all you need. No data to back my hypothesis up. So take it for what it’s worth.
- larodi 2y ago
  My thought on the same guess being - all tokens live in same latent space or in many spaces and each logical units train separate of each other…?
- hadlock 2y ago
  This is where I was headed but I think you said it better. Some kind of executive process monitoring the situation, the random stream of consciousness and the actual output. Looping back around to outdated psychology you have the ego which is the output (speech), the super ego is the executive process and the id is the <think>internal monologue</think>. This isn't the standard definition of those three but close enough.
whimsicalism 2y ago
> this incomprehensible stream of embedding vectors as natural language explanation, in a way returning to encoder/decoder architecture this is just standard decoding, the stream of vectors is called the k/v cache
easeout 2y ago
Here's a paper your idea reminds me of. https://arxiv.org/abs/2501.19201 https://arxiv.org/abs/2501.19201 It's also so not far from Meta's large concept model idea.
- pona-a 2y ago
  Previous discussion: [41 comments, 166 points] https://news.ycombinator.com/item?id=42919597 https://news.ycombinator.com/item?id=42919597
bloomingkales 2y ago
Once we train models on the chain of thought outputs, next token prediction can solve the halting problem for us (eg, this chain of thinking matches this other chain of thinking).
- psadri 2y ago
  I think that is how human brains work. When we practice, at first we have to be deliberate (thinking slow). Then we “learn” from our own experience and it becomes muscle memory (thinking fast). Of course, it increases the odds we are wrong.
  - bloomingkales 2y ago
    Or worse, we incorrectly overweight the wrong chain of thinking to an irrelevant output (but pragmatically useful output), at scale. For example, xenophobia as a response to economic hardship is the wrong chain of thinking embedded in the larger zeitgeist.
jjk7 2y ago
Comments on a google doc? Nesting in social media comments? Seems like similar concepts. I think there is some potential to improving how LLMs improve and further their own reasoning lines, but I'm no AI mage.
zoogeny 2y ago
I've had an idea since I was a kid which I can share. I was contemplating AI and consciousness generally, probably around the time I read "The Minds I". I reflected on the pop-psychology idea of consciousness and subconsciousness. I thought of each as an independent stream of tokens, like stream of consciousness poetry. But along the stream there were joining points between these two streams, points where the conscious stream was edited by the subconscious stream. You could think of the subconscious stream as performing CRUD like operations on the conscious stream. The conscious stream would act like a buffer of short-term memory while the subconscious stream would act like a buffer of long-term memory. Like, the subconscious has instructions related to long-term goals and the conscious stream has instructions related to short-term goals. You can imagine perception as input being fed into the conscious stream and then edited by the subconscious stream before execution. It seems entirely possible to actually implement this idea in this current day and age. I mean, it was a fever dream as a kid, but now it could be an experiment!
- barrenko 2y ago
  Conscious as subconscious pretending not to be sunconscious, something like that, a thin wrapper. Crud makes sense. Gels closely to buddhism, hell, all religions.
- ForHackernews 2y ago
  Have you read Jaynes' "The Origin of Consciousness in the Breakdown of the Bicameral Mind"?
  - zoogeny 2y ago
    I haven't read the original but I am familiar with the broad stroke view. There are similarities (perhaps vague) in the more recent work of someone like McGilchrist and his The Master and His Emissary (another book which I only have a broad stroke view of). At the time I had this idea I did not know of either of these. I think I was drawing explicitly on the conscious / subconscious vocabulary.
cakealert 2y ago
The problem is that RL is extremely inefficient. It's one thing to use it for fine tuning an LLM to do the chain of thought trick and quite another to do thinking entirely from scratch. The pretrained LLM does a lot of heavy lifting there. And it would have to be RL for your idea to work since there is no "thinking" dataset for a novel token space. There isn't even one for existing LLM token space, but they have the base model to work off of. When the thought is expressed in English, the model already knows the relationships between the tokens in the thought, it's merely repurposing it for a "thinking" application.
- itissid 2y ago
  > The problem is that RL is extremely inefficient. Wait What? That is an odd way of defining it. That's like saying turing machines are inefficient way to solve TSP. You would , at the least, want to define this in terms of complexity or put this into context of domains and observability. RL's by definition is a field that is about finding efficient problems in the domain of choice[1]. There are likely regimes in LLM/LRM learning where RL can be quite efficient, polynomial time even in the state space, we just need to explore and find them. For example you can use Dynamic Programming as a "more" efficient way to solve MDPs[1] because it is polynomial in the state space X Action space. [1]https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf https://web.stanford.edu/class/psych209/Readings/SuttonBarto...
  - cakealert 2y ago
    RL provides very poor training signal for deep learning, an order of magnitude or more worse than supervised learning. Better than nothing of course. What the OP suggested is similar to training a transformer from scratch using RL (ie. no training tokens) towards an objective of steering a pretrained LLM to produce human readable output. It will probably not even converge, and if it does it would take immense compute.
    - pizza 2y ago
      In the case of supervised problem domains, you implicitly make a decision about what is signal, and what is noise, and sure, in that closed setting, supervised learning is much more sample efficient. But I think what we're learning now is that with strong enough base models, 'aha' moments in RL training show that it might be possible to essentially 'squeeze out signal from language itself', giving you far greater breadth of latent knowledge than supervised examples, and letting you train to generalize to far greater horizons than a fixed dataset might allow. In a fascinating way it is rather reminiscent of, well, abiogenesis. This might sound like speculative claptrap if you look at the things the current generation of models are still weak at, but... there's a real chance that there is a very heavy tail to the set of outcomes in the limit.

by pona-a 2y ago

bluechair 2y ago

I had this exact same thought yesterday. I’d go so far as to add one more layer to monitor this one and stop adding layers. My thinking is that this meta awareness is all you need. No data to back my hypothesis up. So take it for what it’s worth.

larodi 2y ago
My thought on the same guess being - all tokens live in same latent space or in many spaces and each logical units train separate of each other…?
hadlock 2y ago
This is where I was headed but I think you said it better. Some kind of executive process monitoring the situation, the random stream of consciousness and the actual output. Looping back around to outdated psychology you have the ego which is the output (speech), the super ego is the executive process and the id is the <think>internal monologue</think>. This isn't the standard definition of those three but close enough.

whimsicalism 2y ago

> this incomprehensible stream of embedding vectors as natural language explanation, in a way returning to encoder/decoder architecture this is just standard decoding, the stream of vectors is called the k/v cache

easeout 2y ago

Here's a paper your idea reminds me of. https://arxiv.org/abs/2501.19201 https://arxiv.org/abs/2501.19201 It's also so not far from Meta's large concept model idea.

pona-a 2y ago
Previous discussion: [41 comments, 166 points] https://news.ycombinator.com/item?id=42919597 https://news.ycombinator.com/item?id=42919597

bloomingkales 2y ago

Once we train models on the chain of thought outputs, next token prediction can solve the halting problem for us (eg, this chain of thinking matches this other chain of thinking).

psadri 2y ago
I think that is how human brains work. When we practice, at first we have to be deliberate (thinking slow). Then we “learn” from our own experience and it becomes muscle memory (thinking fast). Of course, it increases the odds we are wrong.
- bloomingkales 2y ago
  Or worse, we incorrectly overweight the wrong chain of thinking to an irrelevant output (but pragmatically useful output), at scale. For example, xenophobia as a response to economic hardship is the wrong chain of thinking embedded in the larger zeitgeist.

jjk7 2y ago

Comments on a google doc? Nesting in social media comments? Seems like similar concepts. I think there is some potential to improving how LLMs improve and further their own reasoning lines, but I'm no AI mage.

zoogeny 2y ago

I've had an idea since I was a kid which I can share. I was contemplating AI and consciousness generally, probably around the time I read "The Minds I". I reflected on the pop-psychology idea of consciousness and subconsciousness. I thought of each as an independent stream of tokens, like stream of consciousness poetry. But along the stream there were joining points between these two streams, points where the conscious stream was edited by the subconscious stream. You could think of the subconscious stream as performing CRUD like operations on the conscious stream. The conscious stream would act like a buffer of short-term memory while the subconscious stream would act like a buffer of long-term memory. Like, the subconscious has instructions related to long-term goals and the conscious stream has instructions related to short-term goals. You can imagine perception as input being fed into the conscious stream and then edited by the subconscious stream before execution. It seems entirely possible to actually implement this idea in this current day and age. I mean, it was a fever dream as a kid, but now it could be an experiment!

barrenko 2y ago
Conscious as subconscious pretending not to be sunconscious, something like that, a thin wrapper. Crud makes sense. Gels closely to buddhism, hell, all religions.
ForHackernews 2y ago
Have you read Jaynes' "The Origin of Consciousness in the Breakdown of the Bicameral Mind"?
- zoogeny 2y ago
  I haven't read the original but I am familiar with the broad stroke view. There are similarities (perhaps vague) in the more recent work of someone like McGilchrist and his The Master and His Emissary (another book which I only have a broad stroke view of). At the time I had this idea I did not know of either of these. I think I was drawing explicitly on the conscious / subconscious vocabulary.

cakealert 2y ago

The problem is that RL is extremely inefficient. It's one thing to use it for fine tuning an LLM to do the chain of thought trick and quite another to do thinking entirely from scratch. The pretrained LLM does a lot of heavy lifting there. And it would have to be RL for your idea to work since there is no "thinking" dataset for a novel token space. There isn't even one for existing LLM token space, but they have the base model to work off of. When the thought is expressed in English, the model already knows the relationships between the tokens in the thought, it's merely repurposing it for a "thinking" application.

itissid 2y ago
> The problem is that RL is extremely inefficient. Wait What? That is an odd way of defining it. That's like saying turing machines are inefficient way to solve TSP. You would , at the least, want to define this in terms of complexity or put this into context of domains and observability. RL's by definition is a field that is about finding efficient problems in the domain of choice[1]. There are likely regimes in LLM/LRM learning where RL can be quite efficient, polynomial time even in the state space, we just need to explore and find them. For example you can use Dynamic Programming as a "more" efficient way to solve MDPs[1] because it is polynomial in the state space X Action space. [1]https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf https://web.stanford.edu/class/psych209/Readings/SuttonBarto...
- cakealert 2y ago
  RL provides very poor training signal for deep learning, an order of magnitude or more worse than supervised learning. Better than nothing of course. What the OP suggested is similar to training a transformer from scratch using RL (ie. no training tokens) towards an objective of steering a pretrained LLM to produce human readable output. It will probably not even converge, and if it does it would take immense compute.
  - pizza 2y ago
    In the case of supervised problem domains, you implicitly make a decision about what is signal, and what is noise, and sure, in that closed setting, supervised learning is much more sample efficient. But I think what we're learning now is that with strong enough base models, 'aha' moments in RL training show that it might be possible to essentially 'squeeze out signal from language itself', giving you far greater breadth of latent knowledge than supervised examples, and letting you train to generalize to far greater horizons than a fixed dataset might allow. In a fascinating way it is rather reminiscent of, well, abiogenesis. This might sound like speculative claptrap if you look at the things the current generation of models are still weak at, but... there's a real chance that there is a very heavy tail to the set of outcomes in the limit.