2 ms·

Is language use not inherently shaped by the real world? The site you tried is a tweet generator, not a question answering site. I prompted GPT-3 with "Bears

Is language use not inherently shaped by the real world? The site you tried is a tweet generator, not a question answering site. I prompted GPT-3 with "Bears and beards are different because" and got... "Bears and beards are different because they are not the same thing. Bears are animals. Beards are facial hair. Bears are dangerous. Beards are not. Bears live in the woods. Beards live on your face. Bears eat people. Beards do not." But my original point was mainly that this field is moving fast and the the old school NLG companies (I created one back in the day!) are toast.

hannasanarion 6y ago
There is more to "the real world" than the definition of words, which is the most that you can expect a language model to learn. Yes it is true that bears are animals. No it is not true that, as GPT-3 said, "There aren't any on the ground"
- visarga 6y ago
  Let's just wait a couple of years to see if GPT-3 was any good in applications. Doesn't matter what we think, what matters is if it is viable. It's younger sibling DALL-E is capable of language grounded in images, I expect the next version to be multi-modal as well. On another line of research there's effort to tame the horse (GPT) by attaching a secondary neural net. This can monitor language, topic, style and bias and ensure increased accuracy in tasks by auto-learning good prompts. It would make development of applications much easier because the base model which was super expensive to train can be reused many times while the secondary net is small and fast to train. Other efforts are related to including a search engine on an inner loop, to make the language model able to query large collections. Also, there's an open effort to create a huge text corpus, so far 800GB (The Pile). It improves on the GPT-3 training corpus on some categories that were lacking. I think it's safe to say the article is way off the current research level.

hannasanarion 6y ago

There is more to "the real world" than the definition of words, which is the most that you can expect a language model to learn. Yes it is true that bears are animals. No it is not true that, as GPT-3 said, "There aren't any on the ground"

visarga 6y ago
Let's just wait a couple of years to see if GPT-3 was any good in applications. Doesn't matter what we think, what matters is if it is viable. It's younger sibling DALL-E is capable of language grounded in images, I expect the next version to be multi-modal as well. On another line of research there's effort to tame the horse (GPT) by attaching a secondary neural net. This can monitor language, topic, style and bias and ensure increased accuracy in tasks by auto-learning good prompts. It would make development of applications much easier because the base model which was super expensive to train can be reused many times while the secondary net is small and fast to train. Other efforts are related to including a search engine on an inner loop, to make the language model able to query large collections. Also, there's an open effort to create a huge text corpus, so far 800GB (The Pile). It improves on the GPT-3 training corpus on some categories that were lacking. I think it's safe to say the article is way off the current research level.