4 ms·

There is more to "the real world" than the definition of words, which is the most that you can expect a language model to learn. Yes it is true that bears are

There is more to "the real world" than the definition of words, which is the most that you can expect a language model to learn. Yes it is true that bears are animals. No it is not true that, as GPT-3 said, "There aren't any on the ground"

visarga 6y ago
Let's just wait a couple of years to see if GPT-3 was any good in applications. Doesn't matter what we think, what matters is if it is viable. It's younger sibling DALL-E is capable of language grounded in images, I expect the next version to be multi-modal as well. On another line of research there's effort to tame the horse (GPT) by attaching a secondary neural net. This can monitor language, topic, style and bias and ensure increased accuracy in tasks by auto-learning good prompts. It would make development of applications much easier because the base model which was super expensive to train can be reused many times while the secondary net is small and fast to train. Other efforts are related to including a search engine on an inner loop, to make the language model able to query large collections. Also, there's an open effort to create a huge text corpus, so far 800GB (The Pile). It improves on the GPT-3 training corpus on some categories that were lacking. I think it's safe to say the article is way off the current research level.