4 ms·

As a programmer but without that much of data science background, could someone explain what is the whole hype/breakthrough of GPT-3? I know it generates conten

As a programmer but without that much of data science background, could someone explain what is the whole hype/breakthrough of GPT-3? I know it generates content that makes sense, "HTML+CSS" and some other text based stuff. But in layman terms, what is the huge deal with it?

vsskanth 6y ago
The first two versions generated content based on a prompt that was coherent for a sentence or a paragraph. GPT-3 generates an entire article that's largely coherent (though still lacking nuanced meaning). Each version of GPT has exponentially increasing parameter count. Raw compute seems to be winning out here. Basically for this application they aren't seeming to hit diminishing returns for increasing parameters. Now people are wondering what GPT-4 can do. If you can write human-level coherent articles or if it passes the turing test or something it's gonna trigger an arms race for governments to obtain this capability.
- brunoluiz 6y ago
  Thanks for the explanation ;) > Each version of GPT has exponentially increasing parameter count What are parameters in this context? > Now people are wondering what GPT-4 can do. If you can write human-level coherent articles or if it passes the turing test or something [...] Well, when I first heard about "text generation" last year what I first raised to my friends was "surely this will make fake news even worse". But well, any technology come with a bright and dark side. I try to not be pessimistic in these moments hehe
  - vsskanth 6y ago
    parameters here are just the size of the neural net of the model. Each unit of computation has its own parameter that needs to be fit. Bigger model = more layers or units of computations = more parameters.
- op03 6y ago
  This is like watching parents talking about their kid develop over time and getting excited over their evolving writing ability. Doesn't mean we get Turing or Gandhi at the end of the story. Or that there is any control over what is produced. To produce those 2 nature had to iterate over 100 billion humans.
- nprz 6y ago
  Disagree with the largely coherent statement. The output I've read from GPT-3 is still quite confusing, for example this[0] blog post demonstrates some of it's capabilities. Reading through the article it generates is very confusing. Statements are made and then later contradicted, elements that do not actually exist on the page are referenced. And there just doesn't seem to be an actual cohesive narrative. All the sentences are grammatically correct and make sense on their own, but the thought connecting the sentences making some sort of larger point just isn't there. And this is after the author generated 10 different articles and picked the most intelligible one. [0] https://maraoz.com/2020/07/18/openai-gpt3/ https://maraoz.com/2020/07/18/openai-gpt3/
tiborsaas 6y ago
It's fascinating to think about it. We put into writing almost everything we experience from the world. GPT-3 demonstrates that if the model is big enough, it can produce intelligent answers, much better than GPT-2. It can fail spectacularly, but humans can do that too :) It was also shown that it's scalable, so there's no reason we couldn't make it an order of magnitude bigger if we wanted to. That future system might be a game changer for search and AI. Then we just have to ask great questions like "what is the meaning of life the universe and everything" and watch the loading animation for 7.5 million years. It's probably not that far fetched to think that this is the Fat Man bomb of AI. China, Russia probably already allocated resources to build their own model. The arms race is on.
- s1t5 6y ago
  > GPT-3 demonstrates that if the model is big enough, it can produce intelligent answers You could make the same statement about a lookup table.
  - tiborsaas 6y ago
    And it would be a fine statement :) That table would be huge in size compared to GPT and less general. GPT is way more compressed.
spyder 6y ago
It's doing all these things (basic code generation, arithmetic, function composition) without specifically being trained for any of these tasks. It was basically just trained to predict the next most likely word from a prompt and yet all those interesting things "emerges" from it. And giving it more compute power it still keeps improving so far.
- abernard1 6y ago
  > And giving it more compute power it still keeps improving so far. And giving it more data, it still keeps improving. The extra compute is necessary to compress and train the model. At the end of the day, we've proven if you have more human knowledge in your lookup table, you can generate more things convincingly. Search engines have been doing something similar for a long time, except instead of generating things, they find them. Search engines also have the advantage that a lot of their knowledge does actually have real semantic models, which seems more intelligent to me.

vsskanth 6y ago

The first two versions generated content based on a prompt that was coherent for a sentence or a paragraph. GPT-3 generates an entire article that's largely coherent (though still lacking nuanced meaning). Each version of GPT has exponentially increasing parameter count. Raw compute seems to be winning out here. Basically for this application they aren't seeming to hit diminishing returns for increasing parameters. Now people are wondering what GPT-4 can do. If you can write human-level coherent articles or if it passes the turing test or something it's gonna trigger an arms race for governments to obtain this capability.

brunoluiz 6y ago
Thanks for the explanation ;) > Each version of GPT has exponentially increasing parameter count What are parameters in this context? > Now people are wondering what GPT-4 can do. If you can write human-level coherent articles or if it passes the turing test or something [...] Well, when I first heard about "text generation" last year what I first raised to my friends was "surely this will make fake news even worse". But well, any technology come with a bright and dark side. I try to not be pessimistic in these moments hehe
- vsskanth 6y ago
  parameters here are just the size of the neural net of the model. Each unit of computation has its own parameter that needs to be fit. Bigger model = more layers or units of computations = more parameters.
op03 6y ago
This is like watching parents talking about their kid develop over time and getting excited over their evolving writing ability. Doesn't mean we get Turing or Gandhi at the end of the story. Or that there is any control over what is produced. To produce those 2 nature had to iterate over 100 billion humans.
nprz 6y ago
Disagree with the largely coherent statement. The output I've read from GPT-3 is still quite confusing, for example this[0] blog post demonstrates some of it's capabilities. Reading through the article it generates is very confusing. Statements are made and then later contradicted, elements that do not actually exist on the page are referenced. And there just doesn't seem to be an actual cohesive narrative. All the sentences are grammatically correct and make sense on their own, but the thought connecting the sentences making some sort of larger point just isn't there. And this is after the author generated 10 different articles and picked the most intelligible one. [0] https://maraoz.com/2020/07/18/openai-gpt3/ https://maraoz.com/2020/07/18/openai-gpt3/

tiborsaas 6y ago

It's fascinating to think about it. We put into writing almost everything we experience from the world. GPT-3 demonstrates that if the model is big enough, it can produce intelligent answers, much better than GPT-2. It can fail spectacularly, but humans can do that too :) It was also shown that it's scalable, so there's no reason we couldn't make it an order of magnitude bigger if we wanted to. That future system might be a game changer for search and AI. Then we just have to ask great questions like "what is the meaning of life the universe and everything" and watch the loading animation for 7.5 million years. It's probably not that far fetched to think that this is the Fat Man bomb of AI. China, Russia probably already allocated resources to build their own model. The arms race is on.

s1t5 6y ago
> GPT-3 demonstrates that if the model is big enough, it can produce intelligent answers You could make the same statement about a lookup table.
- tiborsaas 6y ago
  And it would be a fine statement :) That table would be huge in size compared to GPT and less general. GPT is way more compressed.

spyder 6y ago

It's doing all these things (basic code generation, arithmetic, function composition) without specifically being trained for any of these tasks. It was basically just trained to predict the next most likely word from a prompt and yet all those interesting things "emerges" from it. And giving it more compute power it still keeps improving so far.

abernard1 6y ago
> And giving it more compute power it still keeps improving so far. And giving it more data, it still keeps improving. The extra compute is necessary to compress and train the model. At the end of the day, we've proven if you have more human knowledge in your lookup table, you can generate more things convincingly. Search engines have been doing something similar for a long time, except instead of generating things, they find them. Search engines also have the advantage that a lot of their knowledge does actually have real semantic models, which seems more intelligent to me.