3 ms·

> I bet if you add up that data it's a lot. Let's Fermi estimate that. A 4k video stream is about 50 megabits/second. Let's say that humans have the equivale

> I bet if you add up that data it's a lot. Let's Fermi estimate that. A 4k video stream is about 50 megabits/second. Let's say that humans have the equivalent of two of those going during waking hours, one for vision and one for everything else. Humans are awake for 18 hours/day, and we'll say a human's training is 'complete' at 25. Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data. There's plenty of reason to think that we don't learn effectively from all of this data (there's lots of redundancy, for example), but at the grossest orders of magnitude you seem to be right that at least in theory we have access to a tremendous amount of data.

huijzer 3y ago
> Let's Fermi estimate that. I couldn't agree more. Fermi estimates are underused. > Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data. And for comparison, GPT-4 is estimated by Bill Dally (NVIDIA) at around 10^12 or 10^13 tokens [1]. Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 50 terabyte [Edited after comment below]. As a side note, I would guess that GPT-4 knows more "things" if you would be able to count them all up. For example, it knows more languages, more facts about cities, more facts about persons. However, people know way more inside their specialization. [1]: https://youtu.be/kLiwvnr4L80?si=77pXmIBmlp8dsSCG&t=349 https://youtu.be/kLiwvnr4L80?si=77pXmIBmlp8dsSCG&t=349
- Majromax 3y ago
  > Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 40 petabyte. Thanks, I was looking for that estimate but could not quickly find it. (Although are you sure about that 'petabyte' figure? I see 1e13 * 5 = 50e13 = 50 TB?) I love these sorts of bulk comparisons, since they let us start to reason about how much humans learn from "each bit of experience" versus LLMs. On one hand humans process a lot of sensory information (most of it by discarding, mind), but on the other humans do a lot of 'active learning' whereby experience is directly related to actions. LLMs don't have the latter, since they're trained on passive feeds of tokens.
rainbowzootsuit 3y ago
Your estimate is using compressed video rates, which might be a argued way to look at the way that the brain processes and fills in presumably "already known" or inferred information. I don't know enough about this subject to make that argument. Uncompressed rates are easy to calculate if wanting to adjust your approximation: 24-bit, 4K UHD @ 60 fps: 24 × 3840 × 2160 × 60 = 11.9 Gbit/s.
mdemare 3y ago
Children are awake 12 hours per day, a 90 minute movie used to fit on a CD-ROM, 8 700MB CD-ROMs times 365 is 2TB per year. Alexander the Great became king aged 20, he seemed trained enough by then, so that's 400TB, 4e14 bytes of data. Pretty unconvinced by this argument, it mostly hinges on the video stream size.
- tomohelix 3y ago
  I think it is an estimate. Yours is on the extreme low end of things. The GP was using 4K as a standin for touch, smell, taste, pain, hunger, pleasure, etc. Those are a bit more complex than just videos and you can't encode those on a CD. Regardless, yours still came out to be an order of magnitude higher than what GPT was trained on. So I guess the original argument makes some sense.
saberience 3y ago
So much of that data is totally useless though. Most of the data we get is visual and I would argue that visual data is some of the least useful data (with respect to volume). Think about the amount of time we're looking at blank colours with nothing to learn from. Once you've seen one wall, one chair, one table etc, there's not much left to know about those things. An encyclopedia though for example is much less data than a few hours of high res video of clouds passing by yet clearly its orders of magnitude information rich.
- jonplackett 3y ago
  Also have to disagree here. You see that one object yes, but you see it from 10000000s of different angles and distances. In different settings. In different lighting conditions. With different arrangements. And you see the commonalities and differences. You poke it. prod it. Hit it. Break it. You listen to it. This is the basis for 'common sense' and I’m pretty sure everything else needs that as the foundation. Go watch a child learning and you'll see a hell of a lot of this going on. They want to play with the same thing over and over and over again. They want to watch the same movie over and over and over again. Or the same song over and over and over and over and over again.
- edgyquant 3y ago
  This just sounds so hilariously wrong start to finish. That any of the data you collect is useless is already a big [citation needed]
ComputerGuru 3y ago
This is completely and fundamentally an incorrect approach from start to finish. The human body - and the human mind - do have electrical and logic components but they are absolutely not digital logic. We do not see in “pixels”. The human mind is an analog process. Analog computing is insanely powerful and exponentially more efficient (time, energy, bandwidth) than digital computing but it is ridiculously hard to pack or compose, difficult to build with, limited in the ability to perform imperative logic with, etc. You cannot compare what the human eyes/brain/body does with analog facilities to its digital “equivalent” and then “math it” from there. Also why trying to replicate the human brain with digital logic (current AI approach) is so insanely expensive.
- feoren 3y ago
  You missed the point of the exercise. Of course it's extremely difficult to compare the two, but the question was: do humans get nearly as much training data as LLMs do? This analysis is good enough to say "actually humans receive much more raw input data than LLMs during their 'training period'." You're concerned with what the brain is doing with all that data compared to LLMs, but that's not the point of the exercise.
  - ComputerGuru 3y ago
    No, you can’t even compare that because the information isn’t packetized. Again, we don’t see in pixels so you can’t just consider sensory input in that manner.
- nobodyandproud 3y ago
  What about using energy cost as a stand-in for the data? The net energy cost of training and maintaining humans vs. training and maintaining (updating and augmenting) AI models. The human body is incredibly efficient but there’s always a larger ramp-up cost.
13of40 3y ago
Nit: The bandwidth of your optic nerve is only about 10 kilobits per second. You think you're seeing in 4K but most of it is synthetic.
- CrazyStat 3y ago
  Academic sources I’ve seen are in the 10Mbps range, e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1564115/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1564115/ (estimating 875kbps for a guinea pig and ~10x that for a human).
- jonplackett 3y ago
  That cannot be right. 10 kilobits per second would only get you some awful telephone quality audio. Even super low res video would be orders of magnitude more than that.

huijzer 3y ago

> Let's Fermi estimate that. I couldn't agree more. Fermi estimates are underused. > Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data. And for comparison, GPT-4 is estimated by Bill Dally (NVIDIA) at around 10^12 or 10^13 tokens [1]. Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 50 terabyte [Edited after comment below]. As a side note, I would guess that GPT-4 knows more "things" if you would be able to count them all up. For example, it knows more languages, more facts about cities, more facts about persons. However, people know way more inside their specialization. [1]: https://youtu.be/kLiwvnr4L80?si=77pXmIBmlp8dsSCG&t=349 https://youtu.be/kLiwvnr4L80?si=77pXmIBmlp8dsSCG&t=349

Majromax 3y ago
> Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 40 petabyte. Thanks, I was looking for that estimate but could not quickly find it. (Although are you sure about that 'petabyte' figure? I see 1e13 * 5 = 50e13 = 50 TB?) I love these sorts of bulk comparisons, since they let us start to reason about how much humans learn from "each bit of experience" versus LLMs. On one hand humans process a lot of sensory information (most of it by discarding, mind), but on the other humans do a lot of 'active learning' whereby experience is directly related to actions. LLMs don't have the latter, since they're trained on passive feeds of tokens.

rainbowzootsuit 3y ago

Your estimate is using compressed video rates, which might be a argued way to look at the way that the brain processes and fills in presumably "already known" or inferred information. I don't know enough about this subject to make that argument. Uncompressed rates are easy to calculate if wanting to adjust your approximation: 24-bit, 4K UHD @ 60 fps: 24 × 3840 × 2160 × 60 = 11.9 Gbit/s.

mdemare 3y ago

Children are awake 12 hours per day, a 90 minute movie used to fit on a CD-ROM, 8 700MB CD-ROMs times 365 is 2TB per year. Alexander the Great became king aged 20, he seemed trained enough by then, so that's 400TB, 4e14 bytes of data. Pretty unconvinced by this argument, it mostly hinges on the video stream size.

tomohelix 3y ago
I think it is an estimate. Yours is on the extreme low end of things. The GP was using 4K as a standin for touch, smell, taste, pain, hunger, pleasure, etc. Those are a bit more complex than just videos and you can't encode those on a CD. Regardless, yours still came out to be an order of magnitude higher than what GPT was trained on. So I guess the original argument makes some sense.

saberience 3y ago

So much of that data is totally useless though. Most of the data we get is visual and I would argue that visual data is some of the least useful data (with respect to volume). Think about the amount of time we're looking at blank colours with nothing to learn from. Once you've seen one wall, one chair, one table etc, there's not much left to know about those things. An encyclopedia though for example is much less data than a few hours of high res video of clouds passing by yet clearly its orders of magnitude information rich.

jonplackett 3y ago
Also have to disagree here. You see that one object yes, but you see it from 10000000s of different angles and distances. In different settings. In different lighting conditions. With different arrangements. And you see the commonalities and differences. You poke it. prod it. Hit it. Break it. You listen to it. This is the basis for 'common sense' and I’m pretty sure everything else needs that as the foundation. Go watch a child learning and you'll see a hell of a lot of this going on. They want to play with the same thing over and over and over again. They want to watch the same movie over and over and over again. Or the same song over and over and over and over and over again.
edgyquant 3y ago
This just sounds so hilariously wrong start to finish. That any of the data you collect is useless is already a big [citation needed]

ComputerGuru 3y ago

This is completely and fundamentally an incorrect approach from start to finish. The human body - and the human mind - do have electrical and logic components but they are absolutely not digital logic. We do not see in “pixels”. The human mind is an analog process. Analog computing is insanely powerful and exponentially more efficient (time, energy, bandwidth) than digital computing but it is ridiculously hard to pack or compose, difficult to build with, limited in the ability to perform imperative logic with, etc. You cannot compare what the human eyes/brain/body does with analog facilities to its digital “equivalent” and then “math it” from there. Also why trying to replicate the human brain with digital logic (current AI approach) is so insanely expensive.

feoren 3y ago
You missed the point of the exercise. Of course it's extremely difficult to compare the two, but the question was: do humans get nearly as much training data as LLMs do? This analysis is good enough to say "actually humans receive much more raw input data than LLMs during their 'training period'." You're concerned with what the brain is doing with all that data compared to LLMs, but that's not the point of the exercise.
- ComputerGuru 3y ago
  No, you can’t even compare that because the information isn’t packetized. Again, we don’t see in pixels so you can’t just consider sensory input in that manner.
nobodyandproud 3y ago
What about using energy cost as a stand-in for the data? The net energy cost of training and maintaining humans vs. training and maintaining (updating and augmenting) AI models. The human body is incredibly efficient but there’s always a larger ramp-up cost.

13of40 3y ago

Nit: The bandwidth of your optic nerve is only about 10 kilobits per second. You think you're seeing in 4K but most of it is synthetic.

CrazyStat 3y ago
Academic sources I’ve seen are in the 10Mbps range, e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1564115/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1564115/ (estimating 875kbps for a guinea pig and ~10x that for a human).
jonplackett 3y ago
That cannot be right. 10 kilobits per second would only get you some awful telephone quality audio. Even super low res video would be orders of magnitude more than that.