3 ms·

I’m not arguing about efficiency though ? Simply saying next token predictors cannot be thought of as actually just thinking about the next token with no long t

I’m not arguing about efficiency though ? Simply saying next token predictors cannot be thought of as actually just thinking about the next token with no long term plan.

wizzwizz4 1y ago
They rebuild the "long term plan" anew for every token: there's no guarantee that the reconstructed plan will remain similar between tokens. That's not how planning normally works. (You can find something like this every time there's this kind of gross inefficiency, which is why I gave the general principle.)
- sailingparrot 1y ago
  > They rebuild the "long term plan" anew for every token Well no, there is attention in the LLM which allows it to look back at it's "internal thought" during the previous tokens. Token T at layer L, can attend to a projection of the hidden states of all tokens < T at L. So its definitely not starting anew at every token and is able to iterate on an existing plan. Its not a perfect mechanism for sure, and there is work to make LLMs able to carry more information forward (e.g. feedback transformers), but they can definitely do some of that today.
  - wizzwizz4 1y ago
    This isn't the same as planning. Consider what happens when tokens from another source are appended.
    - sailingparrot 1y ago
      I don't follow how this relates to what we are discussing. Autoregressive LLMs are able to plan within a single forward pass and are able to look back at their previous reasoning and do not start anew at each token like you said. If you append tokens from another source, like in a turn base conversation, then the LLM will process all the new appended tokens in parallel while still being able to look back at it's previous internal state (and thus past reasoning/planning in latent space) from the already processed tokens, then will adjust the plan based on the new information. What happens to you as a human if you come up with a plan with limited information and new information is provided to you?
      - LelouBil 1y ago
        Not the original person you are replying to, but I wanted to add: Yes, they can plan within a single forward pass like you said, but I still think they "start anew at each token" because they have no state/memory that is not the output. I guess this is differing interpretations of the meaning of "start anew", but personally I would agree that having no internal state and simply looking back at it's previous output to form a new token is "starting anew". But I'm also not well informed about the topic so happy to be corrected.
        nl 1y ago
        Worth noting here for others following that a single forward pass is what generates a single token. It's correct to states the LLM starts anew for each token. The work around for this is to pass the existing plan back into it as part of the context.
        sailingparrot 1y ago
        You are forgetting about attention on the kv-cache, which is the mechanism that allows LLM to not start anew everytime.
        nl 1y ago
        I mean sure, but that is a performance optimization that doesn't really change what is going on. It's still recalculating, just that intermediate steps are cached.
        sailingparrot 1y ago
        Isn't the ability to store past reasoning in an external system to avoid having to do the computation all over again precisely what a memory is though? But mathematically KV-caching, instead of doing prefilling at every token is equivalent, sure. But the important part of my message was the attention. A plan/reasoning made during the forward pass of token 0 can be looked at by subsequent (or parallel if you don’t want to use the cache) passes of token 1,…,n. So you cannot consider token n to be starting from scratch in terms of reasoning/planning as it can reuse what has already been planned in previous tokens. If you think about inference with KV-caching, even though you are right that mathematically it’s just an optimization, it makes this behavior much more easy to reason about: the kv-cache is a store of past internal states, that the model can attend to for subsequent tokens, which allows that subsequent token internal hidden states to be more than just a repetition of what the model already reasoned about in the past.
- HarHarVeryFunny 1y ago
  Actually, due to using causal (masked) attention, new tokens appended to the input don't have any effect on what's calculated internally (the "plan") at earlier positions in the input, and a modern LLM therefore uses a KV cache rather than recalculating at those earlier positions. In other words, the "recalculated" plan will be exactly the same as before, just extended with new planning at the position of each newly appended token.
  - astrange 1y ago
    You can violate the plan in the sampler by making an "unreasonable" choice of next token to sample (eg by raising the temperature.) So if it does stick to the same plan, it's not going to be a very good one.
    - HarHarVeryFunny 1y ago
      Yeah. Karpathy recently referred to LLMs having more "working memory" than a human, apparently referring to these unchanging internal activations as "memory", but it's an odd sort of "working memory" if you can't actually update it to reflect progress on what you are working on, or update per new information (new unexpected token having been sampled).
      - sailingparrot 1y ago
        I think a better mental framework of how those model work is that they keep an history of the state of their "memory" across time. Where humans have a single evolving state of our memory LLMs have access to all the states of their "memories" across time, and while past state can't be changed, the new state can: This is the current token's hidden state, and to form this new state they look both at the history of previous states as well as the new information (last token having been sample, or external token from RAG or whatnot appended to the context). This is how progress is stored.
        HarHarVeryFunny 1y ago
- nl 1y ago
  Right, and this is what "reasoning LLMs" work around by having explicitly labelly "reasoning tokens". This lets them "save" the plan between tokens, so when regenerating the new token it is following the plan.

I’m not arguing about efficiency though ? Simply saying next token predictors cannot be thought of as actually just thinking about the next token with no long term plan.

wizzwizz4 1y ago

They rebuild the "long term plan" anew for every token: there's no guarantee that the reconstructed plan will remain similar between tokens. That's not how planning normally works. (You can find something like this every time there's this kind of gross inefficiency, which is why I gave the general principle.)

sailingparrot 1y ago
> They rebuild the "long term plan" anew for every token Well no, there is attention in the LLM which allows it to look back at it's "internal thought" during the previous tokens. Token T at layer L, can attend to a projection of the hidden states of all tokens < T at L. So its definitely not starting anew at every token and is able to iterate on an existing plan. Its not a perfect mechanism for sure, and there is work to make LLMs able to carry more information forward (e.g. feedback transformers), but they can definitely do some of that today.
- wizzwizz4 1y ago
  This isn't the same as planning. Consider what happens when tokens from another source are appended.
  - sailingparrot 1y ago
    I don't follow how this relates to what we are discussing. Autoregressive LLMs are able to plan within a single forward pass and are able to look back at their previous reasoning and do not start anew at each token like you said. If you append tokens from another source, like in a turn base conversation, then the LLM will process all the new appended tokens in parallel while still being able to look back at it's previous internal state (and thus past reasoning/planning in latent space) from the already processed tokens, then will adjust the plan based on the new information. What happens to you as a human if you come up with a plan with limited information and new information is provided to you?
    - LelouBil 1y ago
      Not the original person you are replying to, but I wanted to add: Yes, they can plan within a single forward pass like you said, but I still think they "start anew at each token" because they have no state/memory that is not the output. I guess this is differing interpretations of the meaning of "start anew", but personally I would agree that having no internal state and simply looking back at it's previous output to form a new token is "starting anew". But I'm also not well informed about the topic so happy to be corrected.
      - nl 1y ago
        Worth noting here for others following that a single forward pass is what generates a single token. It's correct to states the LLM starts anew for each token. The work around for this is to pass the existing plan back into it as part of the context.
        sailingparrot 1y ago
        You are forgetting about attention on the kv-cache, which is the mechanism that allows LLM to not start anew everytime.
        nl 1y ago
        I mean sure, but that is a performance optimization that doesn't really change what is going on. It's still recalculating, just that intermediate steps are cached.
        sailingparrot 1y ago
        Isn't the ability to store past reasoning in an external system to avoid having to do the computation all over again precisely what a memory is though? But mathematically KV-caching, instead of doing prefilling at every token is equivalent, sure. But the important part of my message was the attention. A plan/reasoning made during the forward pass of token 0 can be looked at by subsequent (or parallel if you don’t want to use the cache) passes of token 1,…,n. So you cannot consider token n to be starting from scratch in terms of reasoning/planning as it can reuse what has already been planned in previous tokens. If you think about inference with KV-caching, even though you are right that mathematically it’s just an optimization, it makes this behavior much more easy to reason about: the kv-cache is a store of past internal states, that the model can attend to for subsequent tokens, which allows that subsequent token internal hidden states to be more than just a repetition of what the model already reasoned about in the past.
HarHarVeryFunny 1y ago
Actually, due to using causal (masked) attention, new tokens appended to the input don't have any effect on what's calculated internally (the "plan") at earlier positions in the input, and a modern LLM therefore uses a KV cache rather than recalculating at those earlier positions. In other words, the "recalculated" plan will be exactly the same as before, just extended with new planning at the position of each newly appended token.
- astrange 1y ago
  You can violate the plan in the sampler by making an "unreasonable" choice of next token to sample (eg by raising the temperature.) So if it does stick to the same plan, it's not going to be a very good one.
  - HarHarVeryFunny 1y ago
    Yeah. Karpathy recently referred to LLMs having more "working memory" than a human, apparently referring to these unchanging internal activations as "memory", but it's an odd sort of "working memory" if you can't actually update it to reflect progress on what you are working on, or update per new information (new unexpected token having been sampled).
    - sailingparrot 1y ago
      I think a better mental framework of how those model work is that they keep an history of the state of their "memory" across time. Where humans have a single evolving state of our memory LLMs have access to all the states of their "memories" across time, and while past state can't be changed, the new state can: This is the current token's hidden state, and to form this new state they look both at the history of previous states as well as the new information (last token having been sample, or external token from RAG or whatnot appended to the context). This is how progress is stored.
      - HarHarVeryFunny 1y ago
nl 1y ago
Right, and this is what "reasoning LLMs" work around by having explicitly labelly "reasoning tokens". This lets them "save" the plan between tokens, so when regenerating the new token it is following the plan.