3 ms·

> but the insight is probably stated immediately after it. If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the

> but the insight is probably stated immediately after it. If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. There is no reason to draw the conclusion you've drawn. Furthermore, what LLMs are doing isn't thought.

user43928 1mo ago
I don't get your argument. Let's say that the forward pass that selected "Aha" produces activations that indicate a wrong assumption, and a plausible explanation. It puts learned projections of the activation into the KV Cache and outputs Aha. Both the cached projections and the current Aha token can now influence further activations in an additional Forward pass that the Aha bought the model. At least that's how I thought it works.
- wizzwizz4 1mo ago
  A cache is just a cache. I'm not sure what significance you're ascribing to it.
  - user43928 1mo ago
    What is put in the cache?
    - wizzwizz4 1mo ago
      Things that the software running the model would otherwise recompute, if not for the cache. What special meaning are you assigning to it?
      - user43928 1mo ago
        Beats me how it works, honestly can't wrap my head around it. From what I understand, at position Aha in each layer it's constructing a query based on the current activation and looking at the key of each other token position for that layer, in order to decide how much attention to pay to the value. In this way it attends to the previous values, such as perhaps the incorrect assumption and plausible explanation.
nullc 1mo ago
> If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. Yes they do, they have their KV caches-- it's a pure function of the input tokens, sure but that doesn't prevent it from containing latent 'insight'. LLMs can and do pre-form the tokens they're expecting to output multiple steps in the future. I wouldn't argue that the 'aha' means anything, but the structural argument that it can't that I think you're making isn't sound.
throw310822 1mo ago
Consider this: while the inner state of an LLM (all its activations, residuals stream that is cached in the KV cache) is fully deterministic given its input sequence, the information contained in it IS NOT identical to the information in the input sequence. The reason is obvious: the LLM itself contains an enormous amount of information in its parameters and it transfers it to its residuals stream at each forward pass. In other words: the final state given the two input sequences (where NT stands for "null token"): <problem-prompt> [NT] and <problem-prompt> [NT] [NT] [NT] [NT] [NT] [NT] [NT] [NT] is not the same, and at each forward pass the LLM keeps working on the solution even if the input tokens provide absolutely no further information. If this is correct, then there is no need for the model to have already verbalized the key elements that drive the "aha" moment, so no need for the "aha" to appear after a full explanation.

user43928 1mo ago

I don't get your argument. Let's say that the forward pass that selected "Aha" produces activations that indicate a wrong assumption, and a plausible explanation. It puts learned projections of the activation into the KV Cache and outputs Aha. Both the cached projections and the current Aha token can now influence further activations in an additional Forward pass that the Aha bought the model. At least that's how I thought it works.

wizzwizz4 1mo ago
A cache is just a cache. I'm not sure what significance you're ascribing to it.
- user43928 1mo ago
  What is put in the cache?
  - wizzwizz4 1mo ago
    Things that the software running the model would otherwise recompute, if not for the cache. What special meaning are you assigning to it?
    - user43928 1mo ago
      Beats me how it works, honestly can't wrap my head around it. From what I understand, at position Aha in each layer it's constructing a query based on the current activation and looking at the key of each other token position for that layer, in order to decide how much attention to pay to the value. In this way it attends to the previous values, such as perhaps the incorrect assumption and plausible explanation.

nullc 1mo ago

> If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. Yes they do, they have their KV caches-- it's a pure function of the input tokens, sure but that doesn't prevent it from containing latent 'insight'. LLMs can and do pre-form the tokens they're expecting to output multiple steps in the future. I wouldn't argue that the 'aha' means anything, but the structural argument that it can't that I think you're making isn't sound.

throw310822 1mo ago

Consider this: while the inner state of an LLM (all its activations, residuals stream that is cached in the KV cache) is fully deterministic given its input sequence, the information contained in it IS NOT identical to the information in the input sequence. The reason is obvious: the LLM itself contains an enormous amount of information in its parameters and it transfers it to its residuals stream at each forward pass. In other words: the final state given the two input sequences (where NT stands for "null token"): <problem-prompt> [NT] and <problem-prompt> [NT] [NT] [NT] [NT] [NT] [NT] [NT] [NT] is not the same, and at each forward pass the LLM keeps working on the solution even if the input tokens provide absolutely no further information. If this is correct, then there is no need for the model to have already verbalized the key elements that drive the "aha" moment, so no need for the "aha" to appear after a full explanation.