4 ms·

> In context learning is wonderful because it means you can "train" a LLM at run time The “learning” in “In context learning” is not the same concept as learni

> In context learning is wonderful because it means you can "train" a LLM at run time The “learning” in “In context learning” is not the same concept as learning during the network “training”. The former is loosely named and does not have any impact on what’s actually “learned” by the model. The LLM can, by nature, contradict its training data and whatever you/it prompt(s), and incorporates randomness when completing your prompts, hence the term “stochastic”. Nevertheless, it’s useful to have bigger context window if it works well.

nl 3y ago
> The “learning” in “In context learning” is not the same concept as learning during the network “training”. The former is loosely named and does not have any impact on what’s actually “learned” by the model. Actually (and very surprisingly!) it is related. See the "Similarity of Attention Output Updates" and "Similarity of Attention Map" metrics in https://arxiv.org/abs/2212.10559 https://arxiv.org/abs/2212.10559 (linked above) where they show that in-context learning actually changes the attention map in a similar way to traditional fine tuning(!!). Obviously it can't modify the weights of the model on disk, but after ICL new passes through the model are treated quite similarly to how they would be if it has been fine tuned (again: this is astonishing!)
- baobabKoodaa 3y ago
  Whoa. How is that possible?
  - bckr 3y ago
    Yeah, what is being stored that is available between inferences?
    - mike_hearn 3y ago
      Nothing, if by "inference" you mean single API call. The underlying algorithm starts (very roughly) by processing the prompt into a set of vectors, which are then used repeatedly to generate tokens one after the other. What's meant here is that the act of generating and using those prompt vectors has an effect sort of like regular training. It doesn't mean anything is being written to disk afterwards, though of course you could write those vectors to disk and re-use them. It just doesn't make much sense at the moment. Doing that isn't enough to give you infinite context windows.
  - sharepower123 3y ago
    An hypothesis is that in the latent space there is something like a branching factor that can be used in context learning to select the main tree for others layers. So a LLM is able to have the knowledge of many specialized smallers LLM and select the appropriate one by way of giving values to the branching factor. The branching factor could be some combination of attention layers operating in the latent space of previous layers.
  - meh8881 3y ago
    Remember the ancient “king - man = queen” word vectors? This is just that but produced by a model rather than manually adding vectors and finding the nearest one.
  - kgwgk 3y ago
    Depending on what one understands by “new passes through the model” it may not be.
  - deleted 3y ago
    [deleted]
- cubefox 3y ago
  For LLMs there are at least three different ways of "learning": - Pre-training (for text prediction, using unsupervised learning) - Fine-tuning (e.g. to follow instructions or to reject certain queries, using supervised and/or reinforcement learning) (optional) - in-context learning (using "few-shot prompts" as examples) Now the last two can have similar effects. For example, you might fine-tune a foundation (only pre-trained) model to follow instructions, or you don't and instead just modify your prompt such that it looks like a dialogue between a human and a helpful chat assistant. But neither can replace the extensive pre-training phase, which is what gives the model all its intelligence. One other disanalogy between fine-tuning and in-context learning appears to be that the model can't exactly remember the data it was fine-tuned with, while it "knows" exactly everything in its context window. That is its working memory, so to speak.
- numeri 3y ago
  This paper is honestly fairly weak, if you look at it closely. The theoretical justifications rely heavily on approximating attention with linear attention, and really just show that linear attention could be written as (A + A_delta)x, then try to imply (not show – they have now theoretical justification for this next step) that A_delta is a gradient. One could make the same implication about almost any matrix-vector multiplication, by splitting the matrix into two pieces. The empirical studies are fairly weak as well – none of them suggest that there is actually any gradient descent occurring during in-context learning, only that if you fine-tune a model's attention parameters (and no other parameters) for a set of examples, its internal representations and attention patterns are slightly more similar (and it's really not by much!) to those of in-context learning than before fine-tuning. Is it really that surprising? You could most likely take any two techniques that optimize a model towards some target, artificial restrict both of them to only apply to the same set of parameters, use both, and find increased similarity of the two optimized models compared to the non-optimized model. None of that would be what the paper claims, i.e., a shared mechanism of gradient descent, just a similarity in outcome – and we all already knew that ICL and fine-tuning have similar results!
Robotbeat 3y ago
It’s not literally the same as the base model training, but you ARE literally training feed forward networks when you do this in the prompt. That’s how transformers work.

nl 3y ago

> The “learning” in “In context learning” is not the same concept as learning during the network “training”. The former is loosely named and does not have any impact on what’s actually “learned” by the model. Actually (and very surprisingly!) it is related. See the "Similarity of Attention Output Updates" and "Similarity of Attention Map" metrics in https://arxiv.org/abs/2212.10559 https://arxiv.org/abs/2212.10559 (linked above) where they show that in-context learning actually changes the attention map in a similar way to traditional fine tuning(!!). Obviously it can't modify the weights of the model on disk, but after ICL new passes through the model are treated quite similarly to how they would be if it has been fine tuned (again: this is astonishing!)

baobabKoodaa 3y ago
Whoa. How is that possible?
- bckr 3y ago
  Yeah, what is being stored that is available between inferences?
  - mike_hearn 3y ago
    Nothing, if by "inference" you mean single API call. The underlying algorithm starts (very roughly) by processing the prompt into a set of vectors, which are then used repeatedly to generate tokens one after the other. What's meant here is that the act of generating and using those prompt vectors has an effect sort of like regular training. It doesn't mean anything is being written to disk afterwards, though of course you could write those vectors to disk and re-use them. It just doesn't make much sense at the moment. Doing that isn't enough to give you infinite context windows.
- sharepower123 3y ago
  An hypothesis is that in the latent space there is something like a branching factor that can be used in context learning to select the main tree for others layers. So a LLM is able to have the knowledge of many specialized smallers LLM and select the appropriate one by way of giving values to the branching factor. The branching factor could be some combination of attention layers operating in the latent space of previous layers.
- meh8881 3y ago
  Remember the ancient “king - man = queen” word vectors? This is just that but produced by a model rather than manually adding vectors and finding the nearest one.
- kgwgk 3y ago
  Depending on what one understands by “new passes through the model” it may not be.
- deleted 3y ago
  [deleted]
cubefox 3y ago
For LLMs there are at least three different ways of "learning": - Pre-training (for text prediction, using unsupervised learning) - Fine-tuning (e.g. to follow instructions or to reject certain queries, using supervised and/or reinforcement learning) (optional) - in-context learning (using "few-shot prompts" as examples) Now the last two can have similar effects. For example, you might fine-tune a foundation (only pre-trained) model to follow instructions, or you don't and instead just modify your prompt such that it looks like a dialogue between a human and a helpful chat assistant. But neither can replace the extensive pre-training phase, which is what gives the model all its intelligence. One other disanalogy between fine-tuning and in-context learning appears to be that the model can't exactly remember the data it was fine-tuned with, while it "knows" exactly everything in its context window. That is its working memory, so to speak.
numeri 3y ago
This paper is honestly fairly weak, if you look at it closely. The theoretical justifications rely heavily on approximating attention with linear attention, and really just show that linear attention could be written as (A + A_delta)x, then try to imply (not show – they have now theoretical justification for this next step) that A_delta is a gradient. One could make the same implication about almost any matrix-vector multiplication, by splitting the matrix into two pieces. The empirical studies are fairly weak as well – none of them suggest that there is actually any gradient descent occurring during in-context learning, only that if you fine-tune a model's attention parameters (and no other parameters) for a set of examples, its internal representations and attention patterns are slightly more similar (and it's really not by much!) to those of in-context learning than before fine-tuning. Is it really that surprising? You could most likely take any two techniques that optimize a model towards some target, artificial restrict both of them to only apply to the same set of parameters, use both, and find increased similarity of the two optimized models compared to the non-optimized model. None of that would be what the paper claims, i.e., a shared mechanism of gradient descent, just a similarity in outcome – and we all already knew that ICL and fine-tuning have similar results!

Robotbeat 3y ago

It’s not literally the same as the base model training, but you ARE literally training feed forward networks when you do this in the prompt. That’s how transformers work.