5 ms·

ML and NLP surveys are outdated basically the minute the author hits send nowadays. With GPT-3 all of these old school NLG companies seem quaint. Much of their

ML and NLP surveys are outdated basically the minute the author hits send nowadays. With GPT-3 all of these old school NLG companies seem quaint. Much of their functionality can be replicated in minutes with GPT-3, even by non-technical users familiar with prompt design. As soon as OpenAI offer fine tuning OOTB (coming soon) NLP/G will be all but solved for 99% of use cases.

PedroBatista 6y ago
Flying cars buddy! you just wait..
- picodguyo 6y ago
  I know, I know. But I don't say this lightly. I've worked in NLP/G for nearly 15 years and after spending the last 6 months working with GPT-3 I feel the writing is on the wall.
  - Saad_M 6y ago
    But it isn’t. I am still actively involved in NLG both professionally and academically and neural NLG systems have great promise but are still far from actively delivering tangible solutions to areas such data-to-text NLG. Inaccuracy, hallucinations are still highly problematic.
polm23 6y ago
I don't want to suggest you shot off a random comment without actually reading the linked survey. But it spends a whole section discussing that, and why it doesn't see commercial use. Here's a particularly relevant bit: > Sometimes the results produced by GPT-2 and its ilk are quite startling in their apparent authenticity. More often than not they are just a bit off. And sometimes they are just gibberish. As is widely acknowledged, neural text generation as it stands today has a significant problem: driven as it is by information that is ultimately about language use, rather than directly about the real world, it roams untethered to the truth. While the output of such a process might be good enough for the presidential teleprompter, it would not cut it if you want the hard facts about how your pension fund is performing. So, at least for the time being, nobody who develops commercial applications of NLG technology is going to rely on this particular form of that technology.
- picodguyo 6y ago
  I agree with the author on GPT-2. But GPT-3, which became available shortly after this was published, is quite a bit more powerful and there are many commercial applications being built on it now.
  - leereeves 6y ago
    Even GPT-3 knows nothing about the real world; it's merely trained to repeat the words that most often followed the prompt in its training data. That's obviously not useful for news...if a fact is in the training data, it's not news. It's not useful for "hard facts about how your pension fund is performing" unless you want to know how it performed a long time ago. But I agree there are some applications it is useful for, like education.
    - FeepingCreature 6y ago
      > Even GPT-3 knows nothing about the real world; it's merely trained to repeat the words that most often followed the prompt in its training data. I don't know why that would imply that it knows nothing about the real world, unless the data corpus it is trained on likewise bears no relation to reality...
      - nmfisher 6y ago
        > unless the data corpus it is trained on likewise bears no relation to reality It’s trained on Reddit, so I wouldn’t rule that out.
  - probably_wrong 6y ago
    I do not see how GPT-3 could solve the basic architectural problem that the parent comment quotes, namely, that "driven as it is by information that is ultimately about language use, rather than directly about the real world, it roams untethered to the truth". As an experiment I used a GPT-3-powered website [1] to see what GPT-3 has to say about bears, and the first answer was: > "Weird that every day, there are so many cute/funny/entertaining bears to enjoy online but hardly any on the ground." When asked about beards, the first answer has no relation with beards at all: > "If a person doesn’t constantly outwit, outplay, outlast, others, the strong eat the weak." And then there's that time when GPT-3 told someone to kill themselves [2]. While funny and (mostly) grammatically correct, these "thoughts" are nonsense and no amount of extra parameters is going to solve the disconnection between GPT-3 and reality. I imagine you could condition GPT-3 to generate text for a specific piece of data in such a way that guarantees the correctness of its output, but at that point you might as well throw GPT-3 away and write a rule-based system. [1] https://thoughts.sushant-kumar.com/bears https://thoughts.sushant-kumar.com/bears [2] https://www.nabla.com/blog/gpt-3/ https://www.nabla.com/blog/gpt-3/
j-pb 6y ago
"solving" NLP, is AGI complete. GPT-3 is great at superficially correct syntax but breaks at deep / consistent / meaningful semantics.
- yowlingcat 6y ago
  How well can GPT-3 distinguish between human generated and GPT-3 generated?
bulldog13 6y ago
Can you recommend any good tutorials on prompt design for GPT-3 ? Or how to use GPT-3 in general ?
- picodguyo 6y ago
  The main docs are behind a private beta wall, but these are good resources too: http://gptprompts.wikidot.com/ http://gptprompts.wikidot.com/ https://gpttools.com/tutorial_searchQA https://gpttools.com/tutorial_searchQA https://aidungeon.medium.com/world-creation-by-analogy-f26e3791d35f https://aidungeon.medium.com/world-creation-by-analogy-f26e3...

PedroBatista 6y ago

Flying cars buddy! you just wait..

picodguyo 6y ago
I know, I know. But I don't say this lightly. I've worked in NLP/G for nearly 15 years and after spending the last 6 months working with GPT-3 I feel the writing is on the wall.
- Saad_M 6y ago
  But it isn’t. I am still actively involved in NLG both professionally and academically and neural NLG systems have great promise but are still far from actively delivering tangible solutions to areas such data-to-text NLG. Inaccuracy, hallucinations are still highly problematic.

polm23 6y ago

I don't want to suggest you shot off a random comment without actually reading the linked survey. But it spends a whole section discussing that, and why it doesn't see commercial use. Here's a particularly relevant bit: > Sometimes the results produced by GPT-2 and its ilk are quite startling in their apparent authenticity. More often than not they are just a bit off. And sometimes they are just gibberish. As is widely acknowledged, neural text generation as it stands today has a significant problem: driven as it is by information that is ultimately about language use, rather than directly about the real world, it roams untethered to the truth. While the output of such a process might be good enough for the presidential teleprompter, it would not cut it if you want the hard facts about how your pension fund is performing. So, at least for the time being, nobody who develops commercial applications of NLG technology is going to rely on this particular form of that technology.

picodguyo 6y ago
I agree with the author on GPT-2. But GPT-3, which became available shortly after this was published, is quite a bit more powerful and there are many commercial applications being built on it now.
- leereeves 6y ago
  Even GPT-3 knows nothing about the real world; it's merely trained to repeat the words that most often followed the prompt in its training data. That's obviously not useful for news...if a fact is in the training data, it's not news. It's not useful for "hard facts about how your pension fund is performing" unless you want to know how it performed a long time ago. But I agree there are some applications it is useful for, like education.
  - FeepingCreature 6y ago
    > Even GPT-3 knows nothing about the real world; it's merely trained to repeat the words that most often followed the prompt in its training data. I don't know why that would imply that it knows nothing about the real world, unless the data corpus it is trained on likewise bears no relation to reality...
    - nmfisher 6y ago
      > unless the data corpus it is trained on likewise bears no relation to reality It’s trained on Reddit, so I wouldn’t rule that out.
- probably_wrong 6y ago
  I do not see how GPT-3 could solve the basic architectural problem that the parent comment quotes, namely, that "driven as it is by information that is ultimately about language use, rather than directly about the real world, it roams untethered to the truth". As an experiment I used a GPT-3-powered website [1] to see what GPT-3 has to say about bears, and the first answer was: > "Weird that every day, there are so many cute/funny/entertaining bears to enjoy online but hardly any on the ground." When asked about beards, the first answer has no relation with beards at all: > "If a person doesn’t constantly outwit, outplay, outlast, others, the strong eat the weak." And then there's that time when GPT-3 told someone to kill themselves [2]. While funny and (mostly) grammatically correct, these "thoughts" are nonsense and no amount of extra parameters is going to solve the disconnection between GPT-3 and reality. I imagine you could condition GPT-3 to generate text for a specific piece of data in such a way that guarantees the correctness of its output, but at that point you might as well throw GPT-3 away and write a rule-based system. [1] https://thoughts.sushant-kumar.com/bears https://thoughts.sushant-kumar.com/bears [2] https://www.nabla.com/blog/gpt-3/ https://www.nabla.com/blog/gpt-3/

j-pb 6y ago

"solving" NLP, is AGI complete. GPT-3 is great at superficially correct syntax but breaks at deep / consistent / meaningful semantics.

yowlingcat 6y ago
How well can GPT-3 distinguish between human generated and GPT-3 generated?

bulldog13 6y ago

Can you recommend any good tutorials on prompt design for GPT-3 ? Or how to use GPT-3 in general ?

picodguyo 6y ago
The main docs are behind a private beta wall, but these are good resources too: http://gptprompts.wikidot.com/ http://gptprompts.wikidot.com/ https://gpttools.com/tutorial_searchQA https://gpttools.com/tutorial_searchQA https://aidungeon.medium.com/world-creation-by-analogy-f26e3791d35f https://aidungeon.medium.com/world-creation-by-analogy-f26e3...