4 ms·

> no-one can train GPT-3 in any acceptable time except Gwern, who has spent probably as much time with GPT-3 and GPT-2 as any 'amateur' out there, is publicly

> no-one can train GPT-3 in any acceptable time except Gwern, who has spent probably as much time with GPT-3 and GPT-2 as any 'amateur' out there, is publicly out there saying that for most use cases, GPT-3 + creative use of prompts gets you better results than GPT-2 with finetuning. That's an amazing capability that Gwern elaborates on more here: > A new programming paradigm? The GPT-3 neural network is so large a model in terms of power and dataset that it exhibits qualitatively different behavior: you do not apply it to a fixed set of tasks which were in the training dataset, requiring retraining on additional data if one wants to handle a new task (as one would have to retrain GPT-2); instead, you interact with it, expressing any task in terms of natural language descriptions, requests, and examples, tweaking the prompt until it “understands” & it meta-learns the new task based on the high-level abstractions it learned from the pretraining. This is a rather different way of using a DL model, and it’s better to think of it as a new kind of programming, where the prompt is now a “program” which programs GPT-3 to do new things. https://www.gwern.net/GPT-3#prompts-as-programming https://www.gwern.net/GPT-3#prompts-as-programming Almost all the GPT-3 results you see on Twitter are via the OpenAI API - no finetuning, only prompting. That implies... > Without the model, it is not possible to adapt it to any other purpose, unless OpenAI does it for them ... that we're actually very far from plumbing the possible ranges of behavior of GPT-3 with different sorts of prompting. This is a new ballgame folks. The old rules don't quite apply here.

yomly 6y ago
I don't really understand your post (and don't know much about GPT3) are you suggesting that the model is stateful in that it can continue learning from successive prompts? Maybe you can elaborate on this: >instead, you interact with it, expressing any task in terms of natural language descriptions, requests, and examples, tweaking the prompt until it “understands” & it meta-learns the new task based on the high-level abstractions it learned from the pretraining. Or are you suggesting that it has such a deep network of abstractions that once a user starts to map that out, the mileage they can extract back out of the model via prompts is very exciting.
- walleeee 6y ago
  I don't think the claim is that the model is "stateful" in that it continues to learn from prompts. I think it's that the model no longer requires retraining for different situations; instead it has "learned" a set of lower (higher?) level abstractions from which those same (and possibly new) situations can be constructed dynamically from the input prompt.
- smallnamespace 6y ago
  The model is not stateful, but you can emulate state (certainly with GPT-3, but also with other language models) by simply feeding back earlier output. For example, to simulate a chatbot, you start with a prompt. You then successively feed longer and longer chunks of the full chat back to the model, taking incrementally generated lines as the new AI's reply. This is essentially how some of the 'use GPT-2 as a chatbot' front ends work in the world. This is also extended to make things like AI dungeon work: you can force the model to keep context within its attention by providing a good summary in the prompt. To speculate a bit on why this seems to work, these models are massive and have read millions of texts in their corpus. Instead of 'retraining' on text which the model probably has already seen, the prompt is nudging the model to identify where in its on weights its encoded the knowledge before.

yomly 6y ago

I don't really understand your post (and don't know much about GPT3) are you suggesting that the model is stateful in that it can continue learning from successive prompts? Maybe you can elaborate on this: >instead, you interact with it, expressing any task in terms of natural language descriptions, requests, and examples, tweaking the prompt until it “understands” & it meta-learns the new task based on the high-level abstractions it learned from the pretraining. Or are you suggesting that it has such a deep network of abstractions that once a user starts to map that out, the mileage they can extract back out of the model via prompts is very exciting.

walleeee 6y ago
I don't think the claim is that the model is "stateful" in that it continues to learn from prompts. I think it's that the model no longer requires retraining for different situations; instead it has "learned" a set of lower (higher?) level abstractions from which those same (and possibly new) situations can be constructed dynamically from the input prompt.
smallnamespace 6y ago
The model is not stateful, but you can emulate state (certainly with GPT-3, but also with other language models) by simply feeding back earlier output. For example, to simulate a chatbot, you start with a prompt. You then successively feed longer and longer chunks of the full chat back to the model, taking incrementally generated lines as the new AI's reply. This is essentially how some of the 'use GPT-2 as a chatbot' front ends work in the world. This is also extended to make things like AI dungeon work: you can force the model to keep context within its attention by providing a good summary in the prompt. To speculate a bit on why this seems to work, these models are massive and have read millions of texts in their corpus. Instead of 'retraining' on text which the model probably has already seen, the prompt is nudging the model to identify where in its on weights its encoded the knowledge before.