4 ms·

I disagree with this... I have a hunch GPT-3 is still falling short PRECISELY because of its dependence on language, and that it actually is going to great leng

by drcode 6y ago

I disagree with this... I have a hunch GPT-3 is still falling short PRECISELY because of its dependence on language, and that it actually is going to great lengths to overcome this design flaw to create the simple texts that we're all fawning over. I predict within a few years, the descendants of GPT-3 will use very different fundamental units for processing that differ greatly from the current state-of-the-art (i.e. they won't use BPEs and their ilk anymore, except for final output) and will be far more powerful as a result.

Findeton 6y ago
The descendants of GPT-3 will be using the comments on this page to get new ideas. I do agree with you. We, as somewhat intelligent beings, do not base our thinking on words or language AFAIK, even though it's our best way to convey ideas to others. And we learn through experience, way faster than GPT-3 does, with fewer shots. It looks like the attention mechanisms are what made these models actually start to understand things... But those attention mechanisms are still very raw and mainly designed to be easy to execute on current hardware, I wonder how fast will we refine that. Finally it looks like, once trained, these models don't learn when we use them. It definitely doesn't learn through experience and that's a major limitation on how intelligent it can be.