4 ms·
A child learns to talk in about 150M tokens, GPT-4 needs 15T tokens. Humanity needed the total speech of 110B people to evolve to current level, which comes abo
by visarga 2y ago
A child learns to talk in about 150M tokens, GPT-4 needs 15T tokens. Humanity needed the total speech of 110B people to evolve to current level, which comes about 51,395,437,500,000,000,000 tokens, about 3.4 million times more than GPT-4's training run, and 342M times more than a child needs to learn.
That is to show the relative cost between imitation and discovery. Evolution is on the order of 0.3 trillion times slower than catching up by imitation.
- defrost 2y ago> Humanity needed the total speech of 110B people to evolve to current level Seems a loooong stretch to say the least.
- visarga 2y agoI find it interesting to estimate on tokens so we can meaningfully compare people with LLMs.
- defrost 2y agoIt's more the implied dependancy tree ... did Isaac Newton rest on the shoulders of 110 B people to conceive of calculus just as Liebnitz did the same? How much input did the Kalahari bushmen of that time, and their ancestors also, have to that infintesimal leap forwards?
- visarga 2y agoThe dependency tree is not made only of the winning "happy route", it contains all sorts of abandoned branches, otherwise it wouldn't function as an evolutionary system.
- continuational 2y agoThat's apples to oranges. No human ingests that many tokens of speech; individually we learn from far fewer tokens.
- famouswaffles 2y agoNo human brain is the near blank slate an untrained transformer is either. "We only need x tokens of ingested speech" to learn language doesn't hit the same when you have billions of years of the brain baking in an oven to get to that point. But i agree, it's not a direct comparison.
- Jensson 2y ago> No human brain is the near blank slate an untrained transformer is either. Why can't we start out a transformer at a better state and then teach it language in few tokens? Seems like a problem with architecture.
- famouswaffles 2y agoBecause we don't know or understand what a better state would be and so by far the most successful attempts are with architectures with as clean a slate/free of bias as possible. There's nothing wrong with blank slates. It's not a problem, whatever that means. It just is.