3 ms·
It still baffles me that GPT turned out to be more than a glorified markov chain text generator. It seems we’ve actually made it create a model of the world to
by mrfusion 6y ago
It still baffles me that GPT turned out to be more than a glorified markov chain text generator. It seems we’ve actually made it create a model of the world to some degree.
And we kind of just stumbled on the design by throwing massive data and neural networks together?
- Filligree 6y agoIt turns out that brute-force works, and the scaling curve is still not bending. I doubt we'll ever see a GPT-4, because there are known improvements they could make besides just upsizing it further, but that's besides the point. If that curve doesn't bend soon then a 10x larger network would be human-level in many ways. (Well, that is to say. It's actually bending. Upwards.)
- mrfusion 6y agoSo fascinating. I’d love to understand why it’s working so well. I guess no one knows. Wouldn’t gpt4 just be more data and more parameters?
- hntrader 6y agoWhat % of all digitized and reasonably easy-to-access text data did they use to train GPT-3? I'm wondering whether the current limits on GPT-n are computation or data.
- kortex 6y ago> As per the creators, the OpenAI GPT-3 model has been trained about 45 TB text data from multiple sources which include Wikipedia and books. It's about 400 B tokens. Library if Congress is about 40M books, let's say 50K tokens per book, or about 2T tokens. Not necessarily unique. I would say it's plausible that it was a decent percent of the indexed text available, and even more of the unique content. GPT2 was 10B tokens. Do we have 20T tokens available for GPT4? Maybe. But the low hanging fruit are definitely plucked.
- nullc 6y agoYou're made of meat and yet you manage to be more than a glorified markov chain generator. :) (I hope)