5 ms·

> One recent model called Bidirectional Encoder Representations from Transformers (BERT) used 3.3 billion words from English books and Wikipedia articles. Moreo

by jefftk 6y ago

> One recent model called Bidirectional Encoder Representations from Transformers (BERT) used 3.3 billion words from English books and Wikipedia articles. Moreover, during training BERT read this data set not once, but 40 times. To compare, an average child learning to talk might hear 45 million words by age five, 3,000 times fewer than BERT. The human brain is the output of an incredible number of generations of training, representing a vast consumption of energy. Most of the learning that informs the brain happened before this hypothetical five-year-old was even born.

ravi-delia 6y ago
That is a matter of great debate. Although most people's brains wind up settling on basically identical structure over time, case studies with injured or disabled people often display incredible adaptability (ie the occipital lobe in a blind person). If indeed the brain learns most of what it knows from experience, than its (comparatively) low energy consumption would come from greater efficiency. That definitely seems possible; as a basic unit of machine learning the neuron is much more specialized than the transistor, and far slower.
- anaphor 6y ago
  My interpretation of what the parent meant is that the cognitive processes which support language acquisition (which may be mostly domain-general) are already optimized for this task, for which there is a sensitive period during certain years where you need to be exposed to certain inputs, or else you never fully acquire language. See: https://en.wikipedia.org/wiki/Genie_(feral_child) https://en.wikipedia.org/wiki/Genie_(feral_child)
MAXPOOL 6y ago
550–600 million years of hyperparameter tuning and neural architecture search using an evolutionary algorithm is nothing to laugh at. But the actual learning process of a single brain uses very little training examples and is very energy efficient. A brain that uses only 20 watts of power.
- polishdude20 6y ago
  Yeah it's as if we evolved to have a brain that is good at learning itself. We aren't good at any one thing except learning.

by jefftk 6y ago

ravi-delia 6y ago

That is a matter of great debate. Although most people's brains wind up settling on basically identical structure over time, case studies with injured or disabled people often display incredible adaptability (ie the occipital lobe in a blind person). If indeed the brain learns most of what it knows from experience, than its (comparatively) low energy consumption would come from greater efficiency. That definitely seems possible; as a basic unit of machine learning the neuron is much more specialized than the transistor, and far slower.

anaphor 6y ago
My interpretation of what the parent meant is that the cognitive processes which support language acquisition (which may be mostly domain-general) are already optimized for this task, for which there is a sensitive period during certain years where you need to be exposed to certain inputs, or else you never fully acquire language. See: https://en.wikipedia.org/wiki/Genie_(feral_child) https://en.wikipedia.org/wiki/Genie_(feral_child)

MAXPOOL 6y ago

550–600 million years of hyperparameter tuning and neural architecture search using an evolutionary algorithm is nothing to laugh at. But the actual learning process of a single brain uses very little training examples and is very energy efficient. A brain that uses only 20 watts of power.

polishdude20 6y ago
Yeah it's as if we evolved to have a brain that is good at learning itself. We aren't good at any one thing except learning.