3 ms·

GPT-3 was trained on all of those authors.

pmoriarty 6y ago
But many millions of others besides these. So these authors were drowned out in the noise. I'm talking about training on just a relatively small number of authors, so the model can acquire some of their distinctive traits.
- caconym_ 6y ago
  I don't know much about GPT-3, but if you drastically reduce the size of the corpus won't you also drastically reduce the output domain to the point that it's no longer useful? Or is there some way to get it to take whatever "semantic understanding" makes it somewhat convincing, that it's gathered from the full corpus, and ape a particular author's style over it? It's very possible for human authors to imitate style. In fact, if you write a lot of the same kind of thing that you read, you'll find your own style gravitating toward that of whoever you're reading at the time. I just (re)read Franny and Zooey, for instance, and for days I've been fighting Salinger outbreaks in my own prose. Presumably a machine could do the same thing, but I'd guess it's a very hard problem on top of creating something like GPT-3. edit: and this is not even getting into the thematic and semantic richness of the content, which is something good authors pack a lot more of into each word than the blogspam wordsmiths GPT-3 is about to put out of work.
  - pmoriarty 6y ago
    "if you drastically reduce the size of the corpus won't you also drastically reduce the output domain to the point that it's no longer useful?" Usefulness is in the eye of the beholder. A model trained on, say, all the works of Shakespeare might not create something that can pass for Shakespeare's lost work, but might be inspiring, interesting, or have a Shakespearean quality about it in small doses... which could still be very useful for a writer seeking inspiration. Incidentally, all of Shakespeare's works make up a not insignificant amount of data, though obviously it's a lot less than what GPT-3 was trained on. Other prolific authors might also provide enough data to train a model. It's not clear to me at what point the amount of data one uses becomes too little for useful results, and it would surprise me if there was any consensus on this matter, considering the enormous variety in AI techniques, combination of techniques, and parameters that can be used for training and testing, along with all the different uses one might put the models to after they're trained.
    - caconym_ 6y ago
      > Usefulness is in the eye of the beholder. I think the context here is the question of what "kinds" of writing GPT-3 and similar technologies can and can't do as well as humans, so when I talk about "usefulness" I'm really talking about whether GPT-3 can do its thing of generating passable text in response to widely varied prompts. If it can't do that in some domain, then it's not going to replace writers working in that domain. That doesn't mean the output won't be interesting, but it will be far less relevant in the ways that make GPT-3 notable in the first place.
- gwern 6y ago
  "drowned out in the noise" is not really a thing here. Style is just another aspect of text it learns to model and predict. If there are style cues, it makes that part of the prediction and continues it; if there are no style cues, then it just starts somewhere, perhaps with some style common on average... But if there's a decent amount of text clearly linked to a specific author or style, it will learn the quirks, the vocab, the topics, and so on. Shakespare, no problem, of course it can do Shakespeare pastiche - it can do Paul Graham pastiche, it can even do gwern pastiche! Read through the samples in https://www.gwern.net/GPT-3 https://www.gwern.net/GPT-3 and tell me again that everything is just "drowned out in the noise" and GPT-3 is unable to produce anything but a bland average style...

But many millions of others besides these. So these authors were drowned out in the noise. I'm talking about training on just a relatively small number of authors, so the model can acquire some of their distinctive traits.

caconym_ 6y ago
I don't know much about GPT-3, but if you drastically reduce the size of the corpus won't you also drastically reduce the output domain to the point that it's no longer useful? Or is there some way to get it to take whatever "semantic understanding" makes it somewhat convincing, that it's gathered from the full corpus, and ape a particular author's style over it? It's very possible for human authors to imitate style. In fact, if you write a lot of the same kind of thing that you read, you'll find your own style gravitating toward that of whoever you're reading at the time. I just (re)read Franny and Zooey, for instance, and for days I've been fighting Salinger outbreaks in my own prose. Presumably a machine could do the same thing, but I'd guess it's a very hard problem on top of creating something like GPT-3. edit: and this is not even getting into the thematic and semantic richness of the content, which is something good authors pack a lot more of into each word than the blogspam wordsmiths GPT-3 is about to put out of work.
- pmoriarty 6y ago
  "if you drastically reduce the size of the corpus won't you also drastically reduce the output domain to the point that it's no longer useful?" Usefulness is in the eye of the beholder. A model trained on, say, all the works of Shakespeare might not create something that can pass for Shakespeare's lost work, but might be inspiring, interesting, or have a Shakespearean quality about it in small doses... which could still be very useful for a writer seeking inspiration. Incidentally, all of Shakespeare's works make up a not insignificant amount of data, though obviously it's a lot less than what GPT-3 was trained on. Other prolific authors might also provide enough data to train a model. It's not clear to me at what point the amount of data one uses becomes too little for useful results, and it would surprise me if there was any consensus on this matter, considering the enormous variety in AI techniques, combination of techniques, and parameters that can be used for training and testing, along with all the different uses one might put the models to after they're trained.
  - caconym_ 6y ago
    > Usefulness is in the eye of the beholder. I think the context here is the question of what "kinds" of writing GPT-3 and similar technologies can and can't do as well as humans, so when I talk about "usefulness" I'm really talking about whether GPT-3 can do its thing of generating passable text in response to widely varied prompts. If it can't do that in some domain, then it's not going to replace writers working in that domain. That doesn't mean the output won't be interesting, but it will be far less relevant in the ways that make GPT-3 notable in the first place.
gwern 6y ago
"drowned out in the noise" is not really a thing here. Style is just another aspect of text it learns to model and predict. If there are style cues, it makes that part of the prediction and continues it; if there are no style cues, then it just starts somewhere, perhaps with some style common on average... But if there's a decent amount of text clearly linked to a specific author or style, it will learn the quirks, the vocab, the topics, and so on. Shakespare, no problem, of course it can do Shakespeare pastiche - it can do Paul Graham pastiche, it can even do gwern pastiche! Read through the samples in https://www.gwern.net/GPT-3 https://www.gwern.net/GPT-3 and tell me again that everything is just "drowned out in the noise" and GPT-3 is unable to produce anything but a bland average style...