4 ms·

Currently, LLMs start from scratch for each output token, right? Lets say you ask an LLM What makes bananas yellow? And it replies Bananas are yello

by mg 2y ago

Currently, LLMs start from scratch for each output token, right? Lets say you ask an LLM What makes bananas yellow? And it replies Bananas are yellow due to a pigment called bromelain. I would think that the concept of "pigment" and "bromelain" are already somehow activated in the neural net when it outputs "a". Because now it can't change its mind anymore and follow up with "an optical illusion that makes humans perceive every bent object as yellow". So it seems to have already planned ahead to talk about the pigment called bromelain. Would it be possible to capitalize on the work that has already been done when the LLM outputs "a"? Could the state of the neural net be somehow preserved for the next answer?

pbh101 2y ago
The alternative theory is that any word starting with a vowel sound is exceedingly uncommon after ’a’ in its training set, so it doesn’t need to plan ahead, just predict the distribution of the most likely next words and choose. Which is my understanding of how they work and the dynamic at play.
faabian 2y ago
To some degree, attention is already a mechanism to make computations from previous tokens useful later. (You can think of the KV cache as a representation of the text so far and all the models thoughts on it.) And since language models are trained on sequences end-to-end, I think this is likely to happen. Multi-token prediction encourages this behavior explicitly but only for the small n token window you define. That said, there are many works attempting to increase the compute utilization of transformer language models (early exit, mixture of depths) and novel architectures (SSMs etc.).
- jacobsimon 2y ago
  Thanks for highlighting the KV cache, I’ve been wondering the same thing and hadn’t come across that or didn’t remember.
  - edmara 2y ago
    Transformers are still stateless, KV cache is just a compute-saving measure (but otherwise correctly described)
    - jacobsimon 2y ago
      Oh huh. Why not make it stateful, like re-use and compute just the “diff” when you add a new token? Assuming it’s not that easy because each token can affect attention globally. I think I’ve read something about this but I wonder if you could abstract attention to sentence/page levels and then only recalculate the parts that are relevant.
      - edmara 2y ago
        Because attention is all you need. I.E. the KV cache is 'just' a time saving measure because an LLM goes back and calculates those values anyway. (Which is why per-token compute increases exponentially otherwise) You're not wrong that you could make an LLM more stateful. There are plenty of ideas for that but it would a) be far more compute intensive to train and run (especially train) B)be susceptible to all of the issues that RNNs have. C) most importantly, it would almost certainly just converge at scale with transformers. Labs run small scale, internal tests of architectures all the time and most of them basically come to this conclusion and abandon it
avianlyric 2y ago
The output of most LLMs is stochastic. The core LLM is given token, and outputs a set of ranked tokens, with a “confidence”, to go next. Then there’s normally a filtering and search stage, where those ranked token are either feed back into the LLM to get more ranked tokens and used to for a short probability tree. I.e. if we pick the top N-ranked tokens and put them back in, each of those tokens results in a new set of N-ranked tokens. By looking at that tree some basic filtering is done. Such as picking the branch that has the highest summed confidence, or the branch that has the fewest repeated tokens, or the fewest tokens that match with input tokens, or more often some combination of the above, plus a random choice weighed by summed confidences. That how you can give a LLM with complete fixed weights, which is all LLM, the same input multiple times, but get different outputs. So to answer your specific question, it can “change its mind”. Every token produced creates a new opportunity for the stochastic output filters to pick a new path through all the possible outputs.
berkes 2y ago
I always presumed that "a pigment" is one token. I'm a total amateur in this field though.
- ralusek 2y ago
  Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words. If you want to just see the tokens for yourself, though, just enter some text here: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
- sanxiyn 2y ago
  No, it isn't, at least on OpenAI tokenizer.
- jasonjmcghee 2y ago
  what constitutes a token is super unintuitive. my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph" but no, "paragraph" is a single token which falls way outside the "normal" length I see of 3-4 characters In either case, I do consistently see spaces between characters included as part of the token following the space. " paragraph" (10 characters) is the longest token I've seen- and now I wonder what the longest token is
  - jasonjmcghee 2y ago
    New winner " communication" at 14
matthewdgreen 2y ago
This post is interesting: https://clementneo.com/posts/2023/02/11/we-found-an-neuron https://clementneo.com/posts/2023/02/11/we-found-an-neuron
nicklecompte 2y ago
Maybe look at it another way: ask GPT to complete the following Bananas are yellow due to a Bananas are yellow due to an In the first case it might respond Bananas are yellow due to a pigment called bromelain. In the second case it might respond Bananas are yellow due to an organic compound called bromelain, which is a yellow pigment. So in either case GPT could have picked "a" or "an" without any impact on the semantic meaning of its response. In the extreme case, you could see the LLM operating according to a dumb heuristic: The token following "due to" is "a" with 55% probability, "an" with 45% probability. In reality it is of course more sophisticated than this. But this dumb heuristic would explain the behavior. And if you didn't actually include any facts about bromelain in the pretraining data, LLMs absolutely could autocomplete this with something about "an optical illusion." GPT-3 made factual mistakes like that pretty routinely, but I recall it figured out the grammatical rules of "a" and "an." I don't think the concept actually needs to be pre-activated as you said, though I agree with faabian that this "preactivation" probably does happen in some implicit/emergent sense.
HarHarVeryFunny 2y ago
1. Bananas are yellow due to a biochemical process known as carotenogenesis, which involves the synthesis and accumulation of carotenoid pigments. 2. Bananas are yellow due to a specific carotenoid called beta-cryptoxanthin, which gives the fruit its characteristic yellow hue. 3. Bananas are yellow due to a gradual increase in the concentration of carotenoid pigments as the fruit ripens and chlorophyll levels decrease. 4. Bananas are yellow due to a series of enzymatic reactions that convert starch into sugars and break down the green chloroplasts, revealing the underlying yellow carotenoids. 5. Bananas are yellow due to a change in the pH levels within the fruit cells during ripening, which triggers the production of yellow carotenoid pigments. 6. Bananas are yellow due to a genetic trait inherited from their wild ancestors, which enabled the development of carotenoid pigments as a way to attract seed dispersers. 7. Bananas are yellow due to a complex interplay between various plant hormones, such as ethylene and abscisic acid, which regulate the ripening process and pigment formation. 8. Bananas are yellow due to a metabolic shift from chlorophyll synthesis to carotenoid synthesis as the fruit reaches maturity. 9. Bananas are yellow due to a natural defense mechanism that involves the production of carotenoid pigments, which protect the fruit from oxidative stress during ripening. 10. Bananas are yellow due to a evolutionary adaptation that helps the fruit stand out against the green foliage, making it more visible to potential seed dispersers. The output of an LLM is usually randomly sampled from the top few highest probability next token/word predictions, but the model itself has no idea which word the sampler will pick. It presumably has some conceptual plan of what could follow "a", or any of it's other suggestions, but any such plan (high level prediction) is then rethought from scratch once "a" is generated. The model not only can, but has to, change it's mind after each word generated, so this "planning ahead" is very ephemeral - more like a freestyle rapper making it up on the fly than someone thinking deeply about how best to reply and how to express it.
- scoot 2y ago
  > 10. Bananas are yellow due to a evolutionary adaptation Did an LLM really make this basic grammatical error?
  - HarHarVeryFunny 2y ago
wewtyflakes 2y ago
I wonder if this means that these types of models would perform better (or worse?) for languages that do not have this sort of forward-looking grammar.

by mg 2y ago

pbh101 2y ago

The alternative theory is that any word starting with a vowel sound is exceedingly uncommon after ’a’ in its training set, so it doesn’t need to plan ahead, just predict the distribution of the most likely next words and choose. Which is my understanding of how they work and the dynamic at play.

faabian 2y ago

To some degree, attention is already a mechanism to make computations from previous tokens useful later. (You can think of the KV cache as a representation of the text so far and all the models thoughts on it.) And since language models are trained on sequences end-to-end, I think this is likely to happen. Multi-token prediction encourages this behavior explicitly but only for the small n token window you define. That said, there are many works attempting to increase the compute utilization of transformer language models (early exit, mixture of depths) and novel architectures (SSMs etc.).

jacobsimon 2y ago
Thanks for highlighting the KV cache, I’ve been wondering the same thing and hadn’t come across that or didn’t remember.
- edmara 2y ago
  Transformers are still stateless, KV cache is just a compute-saving measure (but otherwise correctly described)
  - jacobsimon 2y ago
    Oh huh. Why not make it stateful, like re-use and compute just the “diff” when you add a new token? Assuming it’s not that easy because each token can affect attention globally. I think I’ve read something about this but I wonder if you could abstract attention to sentence/page levels and then only recalculate the parts that are relevant.
    - edmara 2y ago
      Because attention is all you need. I.E. the KV cache is 'just' a time saving measure because an LLM goes back and calculates those values anyway. (Which is why per-token compute increases exponentially otherwise) You're not wrong that you could make an LLM more stateful. There are plenty of ideas for that but it would a) be far more compute intensive to train and run (especially train) B)be susceptible to all of the issues that RNNs have. C) most importantly, it would almost certainly just converge at scale with transformers. Labs run small scale, internal tests of architectures all the time and most of them basically come to this conclusion and abandon it

avianlyric 2y ago

The output of most LLMs is stochastic. The core LLM is given token, and outputs a set of ranked tokens, with a “confidence”, to go next. Then there’s normally a filtering and search stage, where those ranked token are either feed back into the LLM to get more ranked tokens and used to for a short probability tree. I.e. if we pick the top N-ranked tokens and put them back in, each of those tokens results in a new set of N-ranked tokens. By looking at that tree some basic filtering is done. Such as picking the branch that has the highest summed confidence, or the branch that has the fewest repeated tokens, or the fewest tokens that match with input tokens, or more often some combination of the above, plus a random choice weighed by summed confidences. That how you can give a LLM with complete fixed weights, which is all LLM, the same input multiple times, but get different outputs. So to answer your specific question, it can “change its mind”. Every token produced creates a new opportunity for the stochastic output filters to pick a new path through all the possible outputs.

berkes 2y ago

I always presumed that "a pigment" is one token. I'm a total amateur in this field though.

ralusek 2y ago
Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words. If you want to just see the tokens for yourself, though, just enter some text here: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
sanxiyn 2y ago
No, it isn't, at least on OpenAI tokenizer.
jasonjmcghee 2y ago
what constitutes a token is super unintuitive. my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph" but no, "paragraph" is a single token which falls way outside the "normal" length I see of 3-4 characters In either case, I do consistently see spaces between characters included as part of the token following the space. " paragraph" (10 characters) is the longest token I've seen- and now I wonder what the longest token is
- jasonjmcghee 2y ago
  New winner " communication" at 14

matthewdgreen 2y ago

This post is interesting: https://clementneo.com/posts/2023/02/11/we-found-an-neuron https://clementneo.com/posts/2023/02/11/we-found-an-neuron

nicklecompte 2y ago

Maybe look at it another way: ask GPT to complete the following Bananas are yellow due to a Bananas are yellow due to an In the first case it might respond Bananas are yellow due to a pigment called bromelain. In the second case it might respond Bananas are yellow due to an organic compound called bromelain, which is a yellow pigment. So in either case GPT could have picked "a" or "an" without any impact on the semantic meaning of its response. In the extreme case, you could see the LLM operating according to a dumb heuristic: The token following "due to" is "a" with 55% probability, "an" with 45% probability. In reality it is of course more sophisticated than this. But this dumb heuristic would explain the behavior. And if you didn't actually include any facts about bromelain in the pretraining data, LLMs absolutely could autocomplete this with something about "an optical illusion." GPT-3 made factual mistakes like that pretty routinely, but I recall it figured out the grammatical rules of "a" and "an." I don't think the concept actually needs to be pre-activated as you said, though I agree with faabian that this "preactivation" probably does happen in some implicit/emergent sense.

HarHarVeryFunny 2y ago

1. Bananas are yellow due to a biochemical process known as carotenogenesis, which involves the synthesis and accumulation of carotenoid pigments. 2. Bananas are yellow due to a specific carotenoid called beta-cryptoxanthin, which gives the fruit its characteristic yellow hue. 3. Bananas are yellow due to a gradual increase in the concentration of carotenoid pigments as the fruit ripens and chlorophyll levels decrease. 4. Bananas are yellow due to a series of enzymatic reactions that convert starch into sugars and break down the green chloroplasts, revealing the underlying yellow carotenoids. 5. Bananas are yellow due to a change in the pH levels within the fruit cells during ripening, which triggers the production of yellow carotenoid pigments. 6. Bananas are yellow due to a genetic trait inherited from their wild ancestors, which enabled the development of carotenoid pigments as a way to attract seed dispersers. 7. Bananas are yellow due to a complex interplay between various plant hormones, such as ethylene and abscisic acid, which regulate the ripening process and pigment formation. 8. Bananas are yellow due to a metabolic shift from chlorophyll synthesis to carotenoid synthesis as the fruit reaches maturity. 9. Bananas are yellow due to a natural defense mechanism that involves the production of carotenoid pigments, which protect the fruit from oxidative stress during ripening. 10. Bananas are yellow due to a evolutionary adaptation that helps the fruit stand out against the green foliage, making it more visible to potential seed dispersers. The output of an LLM is usually randomly sampled from the top few highest probability next token/word predictions, but the model itself has no idea which word the sampler will pick. It presumably has some conceptual plan of what could follow "a", or any of it's other suggestions, but any such plan (high level prediction) is then rethought from scratch once "a" is generated. The model not only can, but has to, change it's mind after each word generated, so this "planning ahead" is very ephemeral - more like a freestyle rapper making it up on the fly than someone thinking deeply about how best to reply and how to express it.

scoot 2y ago
> 10. Bananas are yellow due to a evolutionary adaptation Did an LLM really make this basic grammatical error?
- HarHarVeryFunny 2y ago

wewtyflakes 2y ago

I wonder if this means that these types of models would perform better (or worse?) for languages that do not have this sort of forward-looking grammar.