3 ms·

I'm surprised that GPT-3 understands how to write coherently about GPT-3. Common Crawl, the bulk of GPT-3's training data, was collected even before GPT-2 was

by Imnimo 6y ago

I'm surprised that GPT-3 understands how to write coherently about GPT-3. Common Crawl, the bulk of GPT-3's training data, was collected even before GPT-2 was published. Maybe the Wikipedia dump they used was recent enough to contain a few references to GPT-2, but that's it. So if GPT-3 never saw any discussion of GPT-3, and probably almost no discussion of GPT-2, in its training data, how does it know what they are? The prompt text does give a hint that it's a language prediction model, but that hardly seems like it should be enough.

nullc 6y ago
I mean, you could write coherently about GPT3 without ever hearing about it before-- right? You could absolutely write coherently about it with a bit of priming text, ... I mean, that is exactly what you're doing right now. :) Unless you work for open-ai, you've never actually seen GPT3... for all you know for sure it could be some amazing mechanical turk thing postprocessing GPT2 output :)
- Imnimo 6y ago
  I suppose I could have guessed that GPT-2 is a plausible name for the predecessor of GPT-3, and the other predicted predecessor name (PTB) is not really correct. But I'm still surprised that it can hit a believable tone about its own capabilities. I don't think I could have done that if you had asked me before GPT-2 was published (which is essentially what we're doing if GPT-3's training data predates GPT-2). I think I would have under-estimated and it would be immediately clear to a reader familiar with GPT-3 that I didn't actually know what it could or could not do.
  - nullc 6y ago
    > I think I would have under-estimated Keep in mind, the prompt specified that it was the biggest thing since bitcoin and had disruptive potential. Given that you wouldn't write that it sucks. It also did underestimate its own performance: > ... nor do its predicted replies tend to be relevant or even grammatically correct. A prototype that had predicted replies that were convincing in most cases would be much more impressive than the GPT-3 I describe here, although that would probably require many years of training and many iterations of improvements on the model It essentially adopts a common convention for hype articles: A pile of breathless exaggerated hype and then a brief conclusion that suggests that the approach hardly works at the current time, at least if you're reading between the lines. :) I think it's more surprising that it didn't stray further into fantasy. I'd be interested in seeing the author's rejected samples.

by Imnimo 6y ago

nullc 6y ago

I mean, you could write coherently about GPT3 without ever hearing about it before-- right? You could absolutely write coherently about it with a bit of priming text, ... I mean, that is exactly what you're doing right now. :) Unless you work for open-ai, you've never actually seen GPT3... for all you know for sure it could be some amazing mechanical turk thing postprocessing GPT2 output :)

Imnimo 6y ago
I suppose I could have guessed that GPT-2 is a plausible name for the predecessor of GPT-3, and the other predicted predecessor name (PTB) is not really correct. But I'm still surprised that it can hit a believable tone about its own capabilities. I don't think I could have done that if you had asked me before GPT-2 was published (which is essentially what we're doing if GPT-3's training data predates GPT-2). I think I would have under-estimated and it would be immediately clear to a reader familiar with GPT-3 that I didn't actually know what it could or could not do.
- nullc 6y ago
  > I think I would have under-estimated Keep in mind, the prompt specified that it was the biggest thing since bitcoin and had disruptive potential. Given that you wouldn't write that it sucks. It also did underestimate its own performance: > ... nor do its predicted replies tend to be relevant or even grammatically correct. A prototype that had predicted replies that were convincing in most cases would be much more impressive than the GPT-3 I describe here, although that would probably require many years of training and many iterations of improvements on the model It essentially adopts a common convention for hype articles: A pile of breathless exaggerated hype and then a brief conclusion that suggests that the approach hardly works at the current time, at least if you're reading between the lines. :) I think it's more surprising that it didn't stray further into fantasy. I'd be interested in seeing the author's rejected samples.