4 ms·

Not only that, but humans also have access to all of the "training data" of hundreds of millions of years of evolution baked into our brains.

ACCount37 11mo ago
Which must be doing some heavy lifting. Humans ship with all the priors evolution has managed to cram into them. LLMs have to rediscover all of it from scratch just by looking at an awful lot of data.
- hathawsh 11mo ago
  OTOH, all that data is built on patterns that evolved from many years of evolution, so I think the LLM benefits from that evolution also.
  - ACCount37 11mo ago
    Sure, but LLMs are trying to build the algorithms of the human mind backwards, converge on similar functionality based on just some of the inputs and outputs. This isn't an efficient or a lossless process. The fact that they can pull it off to this extent was a very surprising finding.
layer8 11mo ago
I don’t think the amount of data is essential here. The human genome is only around 750 MB, much less than current LLMs, and likely only a small fraction of it determines human intelligence. On the other hand, current LLMs contain immense amounts of factual knowledge that a human newborn carries zero information about. Intelligence likely doesn’t require that much data, and it may be more a question of evolutionary chance. After all, human intelligence is largely (if not exclusively) the result of natural selection from random mutations, with a generation count that’s likely smaller than the number of training iterations of LLMs. We haven’t found a way yet to artificially develop a digital equivalent effectively, and the way we are training neural networks might actually be a dead end here.
- ACCount37 11mo ago
  That just says "low Kolmogorov complexity". All the priors humans ship with can be represented as a relatively compact algorithm. Which gives us no information on computational complexity of running that algorithm, or on what it does exactly. Only that it's small. LLMs don't get that algorithm, so they have to discover certain things the hard way.

ACCount37 11mo ago

Which must be doing some heavy lifting. Humans ship with all the priors evolution has managed to cram into them. LLMs have to rediscover all of it from scratch just by looking at an awful lot of data.

hathawsh 11mo ago
OTOH, all that data is built on patterns that evolved from many years of evolution, so I think the LLM benefits from that evolution also.
- ACCount37 11mo ago
  Sure, but LLMs are trying to build the algorithms of the human mind backwards, converge on similar functionality based on just some of the inputs and outputs. This isn't an efficient or a lossless process. The fact that they can pull it off to this extent was a very surprising finding.

layer8 11mo ago

I don’t think the amount of data is essential here. The human genome is only around 750 MB, much less than current LLMs, and likely only a small fraction of it determines human intelligence. On the other hand, current LLMs contain immense amounts of factual knowledge that a human newborn carries zero information about. Intelligence likely doesn’t require that much data, and it may be more a question of evolutionary chance. After all, human intelligence is largely (if not exclusively) the result of natural selection from random mutations, with a generation count that’s likely smaller than the number of training iterations of LLMs. We haven’t found a way yet to artificially develop a digital equivalent effectively, and the way we are training neural networks might actually be a dead end here.

ACCount37 11mo ago
That just says "low Kolmogorov complexity". All the priors humans ship with can be represented as a relatively compact algorithm. Which gives us no information on computational complexity of running that algorithm, or on what it does exactly. Only that it's small. LLMs don't get that algorithm, so they have to discover certain things the hard way.