3 ms·
Humans are also extensively pretrained by billions of years of evolution, so by starting from scratch GPT is admittedly disadvantaged from the get-go.
by mkaic 3y ago
Humans are also extensively pretrained by billions of years of evolution, so by starting from scratch GPT is admittedly disadvantaged from the get-go.
- georgeg23 3y agoBut the human genome is only 3 gigabytes and the vast majority of that is unlikely to be encoding brain structure.
- retrac 3y ago"Only" 3 gigabytes. The lambda calculus (a system we know is capable of infinite self-complexity, learning, etc. with the right program) can be described in a few hundred bits. And a neural net can be described in the lambda calculus in perhaps a few thousand bits. Also, we have no idea how "compressed" the genome is.
- georgeg23 3y agoBasic structure can be encoded yes (and obviously is given brains have consistent structure), but the weights or parameters, presuming that brains learn via synaptic weights, obviously do not fit in the genome. Compression still must obey information theory.
- canjobear 3y agoTrue, but we don’t know how to recreate that kind of pretraining.