5 ms·

I don't know much about GPT-3, but if you drastically reduce the size of the corpus won't you also drastically reduce the output domain to the point that it's n

I don't know much about GPT-3, but if you drastically reduce the size of the corpus won't you also drastically reduce the output domain to the point that it's no longer useful? Or is there some way to get it to take whatever "semantic understanding" makes it somewhat convincing, that it's gathered from the full corpus, and ape a particular author's style over it? It's very possible for human authors to imitate style. In fact, if you write a lot of the same kind of thing that you read, you'll find your own style gravitating toward that of whoever you're reading at the time. I just (re)read Franny and Zooey, for instance, and for days I've been fighting Salinger outbreaks in my own prose. Presumably a machine could do the same thing, but I'd guess it's a very hard problem on top of creating something like GPT-3. edit: and this is not even getting into the thematic and semantic richness of the content, which is something good authors pack a lot more of into each word than the blogspam wordsmiths GPT-3 is about to put out of work.

pmoriarty 6y ago
"if you drastically reduce the size of the corpus won't you also drastically reduce the output domain to the point that it's no longer useful?" Usefulness is in the eye of the beholder. A model trained on, say, all the works of Shakespeare might not create something that can pass for Shakespeare's lost work, but might be inspiring, interesting, or have a Shakespearean quality about it in small doses... which could still be very useful for a writer seeking inspiration. Incidentally, all of Shakespeare's works make up a not insignificant amount of data, though obviously it's a lot less than what GPT-3 was trained on. Other prolific authors might also provide enough data to train a model. It's not clear to me at what point the amount of data one uses becomes too little for useful results, and it would surprise me if there was any consensus on this matter, considering the enormous variety in AI techniques, combination of techniques, and parameters that can be used for training and testing, along with all the different uses one might put the models to after they're trained.
- caconym_ 6y ago
  > Usefulness is in the eye of the beholder. I think the context here is the question of what "kinds" of writing GPT-3 and similar technologies can and can't do as well as humans, so when I talk about "usefulness" I'm really talking about whether GPT-3 can do its thing of generating passable text in response to widely varied prompts. If it can't do that in some domain, then it's not going to replace writers working in that domain. That doesn't mean the output won't be interesting, but it will be far less relevant in the ways that make GPT-3 notable in the first place.

pmoriarty 6y ago

"if you drastically reduce the size of the corpus won't you also drastically reduce the output domain to the point that it's no longer useful?" Usefulness is in the eye of the beholder. A model trained on, say, all the works of Shakespeare might not create something that can pass for Shakespeare's lost work, but might be inspiring, interesting, or have a Shakespearean quality about it in small doses... which could still be very useful for a writer seeking inspiration. Incidentally, all of Shakespeare's works make up a not insignificant amount of data, though obviously it's a lot less than what GPT-3 was trained on. Other prolific authors might also provide enough data to train a model. It's not clear to me at what point the amount of data one uses becomes too little for useful results, and it would surprise me if there was any consensus on this matter, considering the enormous variety in AI techniques, combination of techniques, and parameters that can be used for training and testing, along with all the different uses one might put the models to after they're trained.

caconym_ 6y ago
> Usefulness is in the eye of the beholder. I think the context here is the question of what "kinds" of writing GPT-3 and similar technologies can and can't do as well as humans, so when I talk about "usefulness" I'm really talking about whether GPT-3 can do its thing of generating passable text in response to widely varied prompts. If it can't do that in some domain, then it's not going to replace writers working in that domain. That doesn't mean the output won't be interesting, but it will be far less relevant in the ways that make GPT-3 notable in the first place.