4 ms·
I'll make a different prediction. GPT-4 will be the last of 'large language models' (at least from a perspective of noteable progress). It will also be when we
by freediver 4y ago
I'll make a different prediction. GPT-4 will be the last of 'large language models' (at least from a perspective of noteable progress).
It will also be when we realize that approach of fitting big data with gradient decent is a dead-end, after finally exhausting nearly all text we can train it on.
We'll than have to backpedal a bit and find another path for achieving A(G)I.
- mcguire 4y agoIf my google-fu is right, ChatGPT was trained on 570GB of data. I asked, "What is the fastest sea mammal?" ChatGPT just produced, "The fastest sea mammal is the peregrine falcon. This bird of prey is able to reach diving speeds of over 240 mph (386 km/h) when hunting for food, making it the fastest sea mammal and one of the fastest animals on the planet. The peregrine falcon is able to achieve such high speeds by diving from great heights and using the force of gravity to accelerate. When hunting, peregrine falcons will spot their prey from above, then tuck their wings and plummet towards the water, accelerating as they go. At the last moment, they will extend their wings and claws to snatch their prey out of the water." (It usually seems to be saying dolphins lately; last week it was saying sailfish about 3/4s of the time.)
- freediver 4y agoMy Kagi-fu says "Be like water, my friend. The size of the data is not important, only the quality. OpenAI curated/filtered 45TB of data to extract those 570GB. Much of the text that we encounter in this world is like the empty chatter of a bird, mere noise that serves no purpose".