3 ms·
Ignoring the “spits out training data” bit which is at best misleading, it’s interesting that you use the word “abstract” here. I recently followed Karpathy’s
by bigyikes 2y ago
Ignoring the “spits out training data” bit which is at best misleading, it’s interesting that you use the word “abstract” here.
I recently followed Karpathy’s GPT-from-scratch tutorial and was fascinated with how clearly you could see the models improving.
With no training, the model spits out uniformly random text. With a bit of training, the model starts generating gibberish. With further training, the model starts recognizing simple character patterns, like putting a consonant after a vowel. Then it learns syllables, and then words, and then sentences. With enough training (and data and parameters, of course) you eventually yield a model like GPT-4 that can write better code than many programmers.
It’s not always that clear cut, but you can clearly observe it moving up the chain of abstraction as the training loss decreases.
What happens when you go even bigger than GPT-4? We have every reason to believe that the models will be able to think more abstractly.
Your “never gonna work” comment flies in the face of exponential curve we find ourselves on.
- ethanwillis 2y agoIf we keep extrapolating eventually GPT will be omniscient. I really can't think of any reason why that wouldn't be the case, given the exponential curve we find ourselves on.
- esafak 2y agoHow do you know you're not on a logistic curve? Don't you think costs and the availability of training data might impose some constraints?
- deleted 2y ago[deleted]
- dragonwriter 2y agoWith real world phenomena that have resource constraints anywhere, a good rule of thumb is: if it looks like an exponential curve, walks like an exponential curve, and quacks like an exponential curve, it’s definitely a logistic curve
- HeatrayEnjoyer 2y agoThe entire universe is training data.
- esafak 2y agoIt is, but we -- humans, and computers -- are limited in our ability to learn from it. We both learn more easily from structured data, like textbooks.
- inference-lord 2y agoI think they're being factitious?
- ethanwillis 2y agoI am. And I think it says a lot about the state of things that many people think I'm being completely serious.
- fire_lake 2y agoThis has the form of a religious belief.
- mistermann 2y agoAnd also non-religious belief...paradoxical!