3 ms·

This is dumb. An individual human takes in more data than modern LLMs do. https://open.substack.com/pub/echoesofid/p/why-llms-struggle-with-basics-too?utm_sour

by flyval 3y ago

This is dumb. An individual human takes in more data than modern LLMs do. https://open.substack.com/pub/echoesofid/p/why-llms-struggle-with-basics-too?utm_source=direct&utm_campaign=post&utm_medium=web https://open.substack.com/pub/echoesofid/p/why-llms-struggle...

majikaja 3y ago
>Just the vision data of a baby’s first year easily adds up to petabytes What encoding is this??
- lostmsu 3y ago
  Uncompressed 2x 8k by 8k 24bpp 24FPS video. Comes at about 500GB per hour.
  - majikaja 3y ago
    Maybe if we store text data as sequences of 10k x 10k PNGs (one for each letter) and add an image recognition layer it would improve LLM perf
  - amscanne 3y ago
    It’s always interesting to think about the exact technological analog for our biological sensors, but I believe that our vision would be way less than that in terms of raw data. We have a super high-res area at the center of vision (the fovea), but the rest is extremely low resolution (but with high movement and light sensitivity). I think we could reasonably say that if an optical nerve has 1mm neurons on average, and they can fire at 250Hz at the most, that’s 250mbps or ~31mb/s per eye of uncompressed data as an upper bound.
    - eastbound 3y ago
      31mb/s = 14GB/hr (bits to bytes). 81TB per year, assuming 16 hours awake per day. Fits snuggly on a large SSD ;)
    - lostmsu 3y ago
      I don't think you correctly calculate bandwidth in this case. You assume 1 bit per neuron per tick, but time when it fires within the tick also matters, and that information is missing from multipliers. Also, there's no reason to use data from optical nerves as input, as it is already precompressed. You should be counting optical receptors instead (120 000 000).
      - amscanne 3y ago
        I don’t think it matters that much. The firing itself takes a couple of milliseconds, and there’s a refactory period of a millisecond. I’m approximating 250hz as the maximum rate of firing. You’re arguing that the neuron can encode more information with the phase (e.g. fire, recover, wait 2ms, fire) but I think information theory tells us the 250hz actually still bounds the information. Maybe there’s a small constant factor, but I don’t think it changes the order. I don’t believe it is precompressed as it hasn’t been processed by the visual cortex yet, no? Aren’t the optical receptors simply an artifact of the “sensor design”? E.g. if the refractory period of an optical receptor is 100x that of the neuron (or you simply need to cover a certain area, as you probably have tons of receptors attached to a single neuron outside the fovea and a small number per neuron inside the fovea), you’d hook up 100 optical receptors per neuron to use its full capacity. I think this is less compression and more combining a bunch of low information channels into a higher information channel. All we really care about here is the amount of information reaching the brain not what your physical eye is capable of receiving, so I think using the nerve makes the most sense. There’s an interesting direct analogy: we don’t really care about the number of CCD sensors in the camera that took the image, we only care about how much information is in the video coming from the camera.
        lostmsu 3y ago
        I disagree with the later point, as unlike camera sensors the cells in question already include trainable parameters for every single one of 100M+ inputs. But it matters little as even with 100x reduction the estimate blows GPT out of the water in the first year, making it very sample inefficient in comparison. As for signal I am a layman in its most extreme here (only mist-like idea about information theory and frequency relationship), but don't the bandwidth limits only apply to fixed rate measurements? E.g. there's basically infinite (sans plank limits) number of values between 4ms and 5ms and as long as the receiver can separate them, they can encode information? To put it in other words, if the neurons can control the impulse peak delay down to a nanosecond, then shouldn't the limit be measured based on 10^9Hz of that control vs 250Hz of max firing rate?
- echelon 3y ago
  > > Just the vision data of a baby’s first year easily adds up to petabytes Just to add to this, the human brain also encodes quite a lot of evolutionary lessons. We didn't have to learn edge detectors.
sebzim4500 3y ago
Blind kids don't though, and they still end up being smarter than GPT-4
- eru 3y ago
  They still take in a lot of sensory data, eg related to touch and proprioception.
  - sebzim4500 3y ago
    True but surely that doesn't come to terabytes of data
    - eru 3y ago
      I would expect similar amount of data (or more) compared to usual visual inputs. After all, your skin is a pretty big organ. And your sense of balance and proprioception in all the joints is quiet a few different channels and pretty high temporal resolution.

by flyval 3y ago

majikaja 3y ago

>Just the vision data of a baby’s first year easily adds up to petabytes What encoding is this??

lostmsu 3y ago
Uncompressed 2x 8k by 8k 24bpp 24FPS video. Comes at about 500GB per hour.
- majikaja 3y ago
  Maybe if we store text data as sequences of 10k x 10k PNGs (one for each letter) and add an image recognition layer it would improve LLM perf
- amscanne 3y ago
  It’s always interesting to think about the exact technological analog for our biological sensors, but I believe that our vision would be way less than that in terms of raw data. We have a super high-res area at the center of vision (the fovea), but the rest is extremely low resolution (but with high movement and light sensitivity). I think we could reasonably say that if an optical nerve has 1mm neurons on average, and they can fire at 250Hz at the most, that’s 250mbps or ~31mb/s per eye of uncompressed data as an upper bound.
  - eastbound 3y ago
    31mb/s = 14GB/hr (bits to bytes). 81TB per year, assuming 16 hours awake per day. Fits snuggly on a large SSD ;)
  - lostmsu 3y ago
    I don't think you correctly calculate bandwidth in this case. You assume 1 bit per neuron per tick, but time when it fires within the tick also matters, and that information is missing from multipliers. Also, there's no reason to use data from optical nerves as input, as it is already precompressed. You should be counting optical receptors instead (120 000 000).
    - amscanne 3y ago
      I don’t think it matters that much. The firing itself takes a couple of milliseconds, and there’s a refactory period of a millisecond. I’m approximating 250hz as the maximum rate of firing. You’re arguing that the neuron can encode more information with the phase (e.g. fire, recover, wait 2ms, fire) but I think information theory tells us the 250hz actually still bounds the information. Maybe there’s a small constant factor, but I don’t think it changes the order. I don’t believe it is precompressed as it hasn’t been processed by the visual cortex yet, no? Aren’t the optical receptors simply an artifact of the “sensor design”? E.g. if the refractory period of an optical receptor is 100x that of the neuron (or you simply need to cover a certain area, as you probably have tons of receptors attached to a single neuron outside the fovea and a small number per neuron inside the fovea), you’d hook up 100 optical receptors per neuron to use its full capacity. I think this is less compression and more combining a bunch of low information channels into a higher information channel. All we really care about here is the amount of information reaching the brain not what your physical eye is capable of receiving, so I think using the nerve makes the most sense. There’s an interesting direct analogy: we don’t really care about the number of CCD sensors in the camera that took the image, we only care about how much information is in the video coming from the camera.
      - lostmsu 3y ago
        I disagree with the later point, as unlike camera sensors the cells in question already include trainable parameters for every single one of 100M+ inputs. But it matters little as even with 100x reduction the estimate blows GPT out of the water in the first year, making it very sample inefficient in comparison. As for signal I am a layman in its most extreme here (only mist-like idea about information theory and frequency relationship), but don't the bandwidth limits only apply to fixed rate measurements? E.g. there's basically infinite (sans plank limits) number of values between 4ms and 5ms and as long as the receiver can separate them, they can encode information? To put it in other words, if the neurons can control the impulse peak delay down to a nanosecond, then shouldn't the limit be measured based on 10^9Hz of that control vs 250Hz of max firing rate?
echelon 3y ago
> > Just the vision data of a baby’s first year easily adds up to petabytes Just to add to this, the human brain also encodes quite a lot of evolutionary lessons. We didn't have to learn edge detectors.

sebzim4500 3y ago

Blind kids don't though, and they still end up being smarter than GPT-4

eru 3y ago
They still take in a lot of sensory data, eg related to touch and proprioception.
- sebzim4500 3y ago
  True but surely that doesn't come to terabytes of data
  - eru 3y ago
    I would expect similar amount of data (or more) compared to usual visual inputs. After all, your skin is a pretty big organ. And your sense of balance and proprioception in all the joints is quiet a few different channels and pretty high temporal resolution.