3 ms·

I don't see the issue here. The exact same definition of "few shot learning" has been used for at least 20 years. Nothing changed with the GPT-3 paper. The def

I don't see the issue here. The exact same definition of "few shot learning" has been used for at least 20 years. Nothing changed with the GPT-3 paper. The definition is something like Given a task, a few shot learner is an algorithm that generalizes well with only a small number of training examples for that task. The same definition is what I'm familiar with from undergrad. Do you know of a different definition that precedes GPT-3?

mjburgess 4y ago
The issue is that the aim is to model the one shot of animals, not some other target . Animals are one shot learners because we're in causally direct sensory-motor contact with reality, such that we can dibiguatw it live. No train/predict system can disambiguate the causal origin of data and so can never be one shot. What they're targeting is a triviality within a system of trivialities and misdiscrubjng it
- mgraczyk 4y ago
  But then you're saying that the definition has always been wrong, which is a very different claim. I personally think that claiming a definition has always been wrong is vacuous. Just substitute the word in your head for something else if you don't like it.
YeGoblynQueenne 4y ago
I reference the GPT-3 preprint because it's the source of the latest twist to the meaning of "few-shot learning". >> Given a task, a few shot learner is an algorithm that generalizes well with only a small number of training examples for that task. I don't know where this definition comes from and I'd prefer if you had a more solid reference than your recollection of your undregraduate years, but it doesn't really matter because what you describe is not what LLMs do. The input prompts to GPT-3 and friends are not "training examples". Training examples are labelled instances of a concept to be learned- that's PAC-Learning. LLM prompts are not examples, and they're not used in training. They're input sequences that an already-trained model completes with a sequence of tokens with maximal probability given the input. That's indeed nothing new, it's how language generation with a language model works, and has always worked. But I don't remember ever hearing anyone referring to the input sequences given to Hidden Markov Models or Probabilistic Context-Free Grammars as "training examples", or the process of generating their completions referred to as "learning", let alone "few-shot learning". And yet, this kind of generation is exactly what LLMs do, too. Except of course LLMs have much smoother, larger models than any HMM or PCFG ever trained. But as the OP is arguing, that's not a qualitative difference, only a quantitative one, and renaming it is misleading. Doubly so if the renaming walks roughshod over long-established terminology, like "few shot", "learning" or "examples". Btw, the OpenAI GPT-3 preprint gives a definition of their "few shot" setting but it's informal, long-wided and overall a vague mess, so it's really no surprise that so much confusion and chaos is generated as a result.
- visarga 4y ago
  > Remarkably, conditioning the model on such an “example-based specification” effectively enables the model to adapt on-the-fly to novel tasks whose distributions of inputs vary significantly from the training distribution. The idea that simply minimizing the negative log loss of a single next-word-prediction objective implies this apparent optimization of many more arbitrary tasks – amounting to a paradigm of “learning” during inference time – is a powerful one, and one that raises many questions. http://ai.stanford.edu/blog/in-context-learning/ http://ai.stanford.edu/blog/in-context-learning/
  - YeGoblynQueenne 4y ago
    Hey man. Give some context to links please. Don't just dump them like that... But thanks for the link, it seems like an interesting read though I'm too tired now to read it (but not for spouting off on HN... i know right). >> Remarkably, conditioning the model on such an “example-based specification” effectively enables the model to adapt on-the-fly to novel tasks whose distributions of inputs vary significantly from the training distribution. That would be indeed interesting if it was the case, but as we have seen in practice, it isn't. For example, GPT-3 is great in addition and subtraction with two- and three-digit numbers, but its performance deteriorates rapidly after that, plus it's bad at multiplication and completely incapable of division [1]. That's exactly what you'd expect of a model trained on a dataset where "easy" arithmetic problems predominate, i.e. the internet, and not at all what you'd expect of a model that can somehow magickally generalise beyond its training distribution without even being trained, and without even retaining any memory of its "examples" (i.e. prompts). In any case, the way to convince me that I'm wrong is to do the good, old-fashioned hard work of proving LLMs' learning ability, like people used to do when it was all hairy stuff like statistical learning theory and VC-dimension and so on. But that's too much hard work and people prefer to poke LLMs with arbitrary prompts instead nowadays. And I agree that it's more fun that way, but at some point you have to give up the fun and try to understand exactly what's going on with a complex system. ____________________ [1] See Figure 3.10 in "Language models are few-shot learners". Strange, I could have sworn there used to be a "large" in the title.

mjburgess 4y ago

The issue is that the aim is to model the one shot of animals, not some other target . Animals are one shot learners because we're in causally direct sensory-motor contact with reality, such that we can dibiguatw it live. No train/predict system can disambiguate the causal origin of data and so can never be one shot. What they're targeting is a triviality within a system of trivialities and misdiscrubjng it

mgraczyk 4y ago
But then you're saying that the definition has always been wrong, which is a very different claim. I personally think that claiming a definition has always been wrong is vacuous. Just substitute the word in your head for something else if you don't like it.

YeGoblynQueenne 4y ago

I reference the GPT-3 preprint because it's the source of the latest twist to the meaning of "few-shot learning". >> Given a task, a few shot learner is an algorithm that generalizes well with only a small number of training examples for that task. I don't know where this definition comes from and I'd prefer if you had a more solid reference than your recollection of your undregraduate years, but it doesn't really matter because what you describe is not what LLMs do. The input prompts to GPT-3 and friends are not "training examples". Training examples are labelled instances of a concept to be learned- that's PAC-Learning. LLM prompts are not examples, and they're not used in training. They're input sequences that an already-trained model completes with a sequence of tokens with maximal probability given the input. That's indeed nothing new, it's how language generation with a language model works, and has always worked. But I don't remember ever hearing anyone referring to the input sequences given to Hidden Markov Models or Probabilistic Context-Free Grammars as "training examples", or the process of generating their completions referred to as "learning", let alone "few-shot learning". And yet, this kind of generation is exactly what LLMs do, too. Except of course LLMs have much smoother, larger models than any HMM or PCFG ever trained. But as the OP is arguing, that's not a qualitative difference, only a quantitative one, and renaming it is misleading. Doubly so if the renaming walks roughshod over long-established terminology, like "few shot", "learning" or "examples". Btw, the OpenAI GPT-3 preprint gives a definition of their "few shot" setting but it's informal, long-wided and overall a vague mess, so it's really no surprise that so much confusion and chaos is generated as a result.

visarga 4y ago
> Remarkably, conditioning the model on such an “example-based specification” effectively enables the model to adapt on-the-fly to novel tasks whose distributions of inputs vary significantly from the training distribution. The idea that simply minimizing the negative log loss of a single next-word-prediction objective implies this apparent optimization of many more arbitrary tasks – amounting to a paradigm of “learning” during inference time – is a powerful one, and one that raises many questions. http://ai.stanford.edu/blog/in-context-learning/ http://ai.stanford.edu/blog/in-context-learning/
- YeGoblynQueenne 4y ago
  Hey man. Give some context to links please. Don't just dump them like that... But thanks for the link, it seems like an interesting read though I'm too tired now to read it (but not for spouting off on HN... i know right). >> Remarkably, conditioning the model on such an “example-based specification” effectively enables the model to adapt on-the-fly to novel tasks whose distributions of inputs vary significantly from the training distribution. That would be indeed interesting if it was the case, but as we have seen in practice, it isn't. For example, GPT-3 is great in addition and subtraction with two- and three-digit numbers, but its performance deteriorates rapidly after that, plus it's bad at multiplication and completely incapable of division [1]. That's exactly what you'd expect of a model trained on a dataset where "easy" arithmetic problems predominate, i.e. the internet, and not at all what you'd expect of a model that can somehow magickally generalise beyond its training distribution without even being trained, and without even retaining any memory of its "examples" (i.e. prompts). In any case, the way to convince me that I'm wrong is to do the good, old-fashioned hard work of proving LLMs' learning ability, like people used to do when it was all hairy stuff like statistical learning theory and VC-dimension and so on. But that's too much hard work and people prefer to poke LLMs with arbitrary prompts instead nowadays. And I agree that it's more fun that way, but at some point you have to give up the fun and try to understand exactly what's going on with a complex system. ____________________ [1] See Figure 3.10 in "Language models are few-shot learners". Strange, I could have sworn there used to be a "large" in the title.