5 ms·

Yet we learning from 18 hours of ultra high resolution video per day (x2), along with 18 hours of reinforcement learning, along with 18 hours of audio and 18 ho

Yet we learning from 18 hours of ultra high resolution video per day (x2), along with 18 hours of reinforcement learning, along with 18 hours of audio and 18 hours of nerve stimulus data. This is assuming we are learning nothing during sleep, which probably isn't true. By the time a person is 21 years old, they have been trained on at least 1 petabyte of data. By two years old, about 125TB of data. It makes LLMs look quite good in comparison.

abetusk 4y ago
By no means rigorous, an estimate for the number of neurons is around 10^10 [0] with an average number of connections estimated at around 8000 [1]. Call 10^10 \approx 2^40 for convenience, and 8000 \approx 2^13, which gives us a 2^53 entropy estimate, or about a petabyte of information as an estimate of what the human brain can store (discounting more exotic theories of memory stored in DNA or some such). [0] https://en.wikipedia.org/wiki/Human_brain#Microanatomy https://en.wikipedia.org/wiki/Human_brain#Microanatomy [1] https://psychology.stackexchange.com/questions/7967/how-many-synapses-in-the-average-human-brain https://psychology.stackexchange.com/questions/7967/how-many...
- edulix 4y ago
  Add to that that our neurons are more complex to point neurons used in typical Artificial neural-nets. A single pyramid neuron in the neocortex might be more comparable to a multilayer neural net. https://www.biorxiv.org/content/10.1101/2021.10.25.465651v1.full.pdf https://www.biorxiv.org/content/10.1101/2021.10.25.465651v1....
  - pmoriarty 4y ago
    Their communication is much more complex too, and neither they nor their connections are static. We don't understand how they work at the subatomic level simply because human understanding of the subatomic world is not complete, but even just at the atomic level a single neuron is massively more complex than anything humans have created. Going up to the molecular level, even that is staggeringly more complex than the incredibly simple abstractions that make up a neural net. Is what happens in the brain at the molecular, atomic, or subatomic levels relevant or necessary to intelligence and consciousness? We just don't know yet, but we do know all of that is far more complex and very different from the simple abstractions that are used for neural nets and LLMs. The back of a napkin calculations in this thread don't even begin to do justice to the tremendous amount of "calculation" or "storage" that happens in the human brain.
- fnordpiglet 4y ago
  Although I’d note this is more like a 1pb stored model.
- deleted 4y ago
  [deleted]
mtlmtlmtlmtl 4y ago
And by far most of that data is completely meaningless. You just can't naïvely compare that with LLMs where all the data is curated and having meaningful content.
- teruakohatu 4y ago
  The majority of the data is a web crawl. It's not curated in the slightest and includes the whole of reddit. In contrast the data we collect through our nervous system is rich and meaningful and far deeper than just raw text. We can even manipulate the environment as we learn to facilitate faster learning eg. pick up a ball and throw it, rather than just watch videos of balls being thrown.
  - z3c0 4y ago
    The web crawling is curation, as even the most mundane entries into the dataset will still have a significant weight on the model, even if it's small. What separates the brain from any statistics-driven model is how well it swims through an ocean of information to find the relevant details, tossing out what's left, and filling in gaps with intuition. Sure, its messy process, and mistakes get made, but that's in stark contrast to the very trim way that ML models run.
tgv 4y ago
> Yet we learning from 18 hours of ultra high resolution video per day (x2) Our object recognition is trained pretty quickly. And we sleep (certainly as a child) more than 6 hours per day. But we don't learn much from just looking at pictures. > 18 hours of reinforcement learning You made that up. > It makes LLMs look quite good in comparison. So, exposing an LLM to a lot of video will make them understand language?
- vkazanov 4y ago
  ...exposing llm to a lot of mostly labelled multimedia content
  - tgv 4y ago
    That's not really how we learn, is it?
  - HPsquared 4y ago
    The world as experienced by humans is unlabelled. We do, however, have instinct.
- lostmsu 4y ago
  The poster above for some reason though encoded video is a good measure, which does not make sense actually. Brain gets raw video. With raw video it is about 100PB per year, > 200TB per day. Even sound alone (uncompressed CD quality stereo) is 3TB/year.
usgroup 4y ago
Blind people learn just fine too. Some deaf/blind people can read braille — they learn fine too. Much less data than you might think. Bare in mind most people can run the brain on 2500kCal/day. And so on.
- SergeAx 4y ago
  > Bare in mind most people can run the brain on 2500kCal/day Which is, in turn, 2900 W·h, or 2.5 times less than one A100 card working round the clock (300W·24h)
  - kristianp 4y ago
    A quick google says that the brain uses about 15W of power. Or 20% of total usage of the person.
    - SergeAx 4y ago
      This is not entirely correct: brain needs oxygen, cooling and toxins removal, which are provided by circulatory, respiratory and secretion systems.
eternauta3k 4y ago
Don't forget that we are trained in an interactive environment.
MagicMoonlight 4y ago
Most of that data is the same all day e.g vision. There’s very little difference between frames. The significant thing is the reinforcement. Without it you have to resort to the openai brainlet style 10tb of text training. If their model had reinforcement built in then you could just plop it in front of a person or on reddit and it would rapidly self-learn on a fraction of the data. Their training model relies on it learning purely based on observations rather than interaction which is inefficient.

abetusk 4y ago

By no means rigorous, an estimate for the number of neurons is around 10^10 [0] with an average number of connections estimated at around 8000 [1]. Call 10^10 \approx 2^40 for convenience, and 8000 \approx 2^13, which gives us a 2^53 entropy estimate, or about a petabyte of information as an estimate of what the human brain can store (discounting more exotic theories of memory stored in DNA or some such). [0] https://en.wikipedia.org/wiki/Human_brain#Microanatomy https://en.wikipedia.org/wiki/Human_brain#Microanatomy [1] https://psychology.stackexchange.com/questions/7967/how-many-synapses-in-the-average-human-brain https://psychology.stackexchange.com/questions/7967/how-many...

edulix 4y ago
Add to that that our neurons are more complex to point neurons used in typical Artificial neural-nets. A single pyramid neuron in the neocortex might be more comparable to a multilayer neural net. https://www.biorxiv.org/content/10.1101/2021.10.25.465651v1.full.pdf https://www.biorxiv.org/content/10.1101/2021.10.25.465651v1....
- pmoriarty 4y ago
  Their communication is much more complex too, and neither they nor their connections are static. We don't understand how they work at the subatomic level simply because human understanding of the subatomic world is not complete, but even just at the atomic level a single neuron is massively more complex than anything humans have created. Going up to the molecular level, even that is staggeringly more complex than the incredibly simple abstractions that make up a neural net. Is what happens in the brain at the molecular, atomic, or subatomic levels relevant or necessary to intelligence and consciousness? We just don't know yet, but we do know all of that is far more complex and very different from the simple abstractions that are used for neural nets and LLMs. The back of a napkin calculations in this thread don't even begin to do justice to the tremendous amount of "calculation" or "storage" that happens in the human brain.
fnordpiglet 4y ago
Although I’d note this is more like a 1pb stored model.
deleted 4y ago
[deleted]

mtlmtlmtlmtl 4y ago

And by far most of that data is completely meaningless. You just can't naïvely compare that with LLMs where all the data is curated and having meaningful content.

teruakohatu 4y ago
The majority of the data is a web crawl. It's not curated in the slightest and includes the whole of reddit. In contrast the data we collect through our nervous system is rich and meaningful and far deeper than just raw text. We can even manipulate the environment as we learn to facilitate faster learning eg. pick up a ball and throw it, rather than just watch videos of balls being thrown.
- z3c0 4y ago
  The web crawling is curation, as even the most mundane entries into the dataset will still have a significant weight on the model, even if it's small. What separates the brain from any statistics-driven model is how well it swims through an ocean of information to find the relevant details, tossing out what's left, and filling in gaps with intuition. Sure, its messy process, and mistakes get made, but that's in stark contrast to the very trim way that ML models run.

tgv 4y ago

> Yet we learning from 18 hours of ultra high resolution video per day (x2) Our object recognition is trained pretty quickly. And we sleep (certainly as a child) more than 6 hours per day. But we don't learn much from just looking at pictures. > 18 hours of reinforcement learning You made that up. > It makes LLMs look quite good in comparison. So, exposing an LLM to a lot of video will make them understand language?

vkazanov 4y ago
...exposing llm to a lot of mostly labelled multimedia content
- tgv 4y ago
  That's not really how we learn, is it?
- HPsquared 4y ago
  The world as experienced by humans is unlabelled. We do, however, have instinct.
lostmsu 4y ago
The poster above for some reason though encoded video is a good measure, which does not make sense actually. Brain gets raw video. With raw video it is about 100PB per year, > 200TB per day. Even sound alone (uncompressed CD quality stereo) is 3TB/year.

usgroup 4y ago

Blind people learn just fine too. Some deaf/blind people can read braille — they learn fine too. Much less data than you might think. Bare in mind most people can run the brain on 2500kCal/day. And so on.

SergeAx 4y ago
> Bare in mind most people can run the brain on 2500kCal/day Which is, in turn, 2900 W·h, or 2.5 times less than one A100 card working round the clock (300W·24h)
- kristianp 4y ago
  A quick google says that the brain uses about 15W of power. Or 20% of total usage of the person.
  - SergeAx 4y ago
    This is not entirely correct: brain needs oxygen, cooling and toxins removal, which are provided by circulatory, respiratory and secretion systems.

eternauta3k 4y ago

Don't forget that we are trained in an interactive environment.

MagicMoonlight 4y ago

Most of that data is the same all day e.g vision. There’s very little difference between frames. The significant thing is the reinforcement. Without it you have to resort to the openai brainlet style 10tb of text training. If their model had reinforcement built in then you could just plop it in front of a person or on reddit and it would rapidly self-learn on a fraction of the data. Their training model relies on it learning purely based on observations rather than interaction which is inefficient.