4 ms·

It is an intelligent argument but it isn't leveraging it's own insight enough. > So then how do humans generalize so well from so little language data? Is “pre

by roenxi 2y ago

It is an intelligent argument but it isn't leveraging it's own insight enough. > So then how do humans generalize so well from so little language data? Is “pre-training” on visual data the secret to our success? No. Because… blind people? What are we doing here? The problem there is that the author making the same mistake they identified earlier. Things like vision or touch are techniques, what matters is the amount of data that is ingested through them. Vision obviously presents vastly more data than text. But a sense of touch is actually vastly more data too, because touch is interacting with the same source of data as vision. It loses colour information and the bandwidth is lower, but there is still a lot more there than text. If the article had estimated how much data was present in bytes it wouldn't have dismissed vision so easily. What matters is there are orders of magnitude more data than all the text on the internet available for training in the real world. Data that can't be polluted by AI blogspam, for that matter.

jcgrillo 2y ago
How many bits of all that sensory information--note that our vision is nearly optimally sensitive--actually make it to the brain? I think there's got to be a large amount of low-effort (structurally baked-in, automatic) filtering, aggregation, and lossy compression happening, otherwise it would just be way too much, right?
- s1artibartfast 2y ago
  While im sure there is a lot of noise filtered out, the opposite is true as well. Much of what we perceive is interpolated and pattern matched filler. I dont know if this is due to source input limitations, or if it is a compression>processing>decompression technique for efficiency. Either way, it does imply that the amount of data desired is more than makes it through the bottleneck.
  - jcgrillo 2y ago
    Given evolution's "modularity/abstraction/engineering principles be damned" approach to problem solving, I'd wager the shape of the bottleneck and every other little detail is actually important. Put another way, evolution writes the worst side-effecting spaghetti code you've ever seen that somehow, (seemingly) miraculously does exactly the right thing, robustly, more efficiently than you can possibly imagine doing it.
    - roenxi 2y ago
      > right thing, robustly, more efficiently than you can possibly imagine doing it. Well... actually evolution doesn't tend to do that well against an engineered process. It has higher bandwidth than human engineers. For example, engineers can move humans around faster and with better energy efficiency [0] than evolution managed directly. We've also figured out more effective ways to organise society (laws & principles) than nature managed (many of our instincts have effects that lead to measurably stupid outcomes, eg, mob forming instincts are just a disaster). There are quite a lot of examples where it turns out engineering > evolution in a quite strict sense. [0] https://en.wikipedia.org/wiki/Energy_efficiency_in_transport https://en.wikipedia.org/wiki/Energy_efficiency_in_transport
      - jcgrillo 2y ago
        Evolution made the engineers, though. And the politicians. We are just as much a "part of nature" as, say, rodents.
      - skydhash 2y ago
        > There are quite a lot of examples where it turns out engineering > evolution in a quite strict sense. Only if you're considering human agents more important than anything else. When considered the system as a whole, nature seems very efficient.
- andrewflnr 2y ago
  Building the noise filter is part of the learning. I don't see any reason to believe that it's "structural". Babies aren't known for their stellar information processing.
- roenxi 2y ago
  Practically it has to be higher than the bitrate of audio-visual data that is presented on a computer. Call that 1MBps for a video stream (way under-calling the amount of data human vision reports I would suggest). That'd put a lower cap of around 50 GB/day of new data, 20 TB/year. Of course, computers can train with more than 2 eyes with one location and perspective. We aren't anywhere near the data cap with current training. Although to be fair I do suspect that most of that data is repetitive, boring and of little use. In my opinion some sort of check for novel data is probably going to be the next big breakthrough in machine learning.
  - beeflet 2y ago
    > (way under-calling the amount of data human vision reports I would suggest) It doesn't matter because the amount of meaningful data for learning is accessible at a lower resolution. If someone is born with bad vision such that they effectively see at 1/2 or 1/4th of the resolution of a normal person, It's not like they will grow up to be stupid (as long as they can sit in the front of the class to see the chalkboard). In order for visual quality to impair learning it would need to be pretty bad such that you couldn't make out objects or symbols at a reasonable range.
whiplash451 2y ago
Mentioning the blind as evidence that there is no pre-training based on vision is broken in my opinion. It could very well be that humans do multimodal pretraining (including vision), landing babies with a pre-trained brains that works even without vision. A machine implementation of this is pre-training a self-driving car on vision + lidar then doing real-time inference on vision only.

by roenxi 2y ago

jcgrillo 2y ago

How many bits of all that sensory information--note that our vision is nearly optimally sensitive--actually make it to the brain? I think there's got to be a large amount of low-effort (structurally baked-in, automatic) filtering, aggregation, and lossy compression happening, otherwise it would just be way too much, right?

s1artibartfast 2y ago
While im sure there is a lot of noise filtered out, the opposite is true as well. Much of what we perceive is interpolated and pattern matched filler. I dont know if this is due to source input limitations, or if it is a compression>processing>decompression technique for efficiency. Either way, it does imply that the amount of data desired is more than makes it through the bottleneck.
- jcgrillo 2y ago
  Given evolution's "modularity/abstraction/engineering principles be damned" approach to problem solving, I'd wager the shape of the bottleneck and every other little detail is actually important. Put another way, evolution writes the worst side-effecting spaghetti code you've ever seen that somehow, (seemingly) miraculously does exactly the right thing, robustly, more efficiently than you can possibly imagine doing it.
  - roenxi 2y ago
    > right thing, robustly, more efficiently than you can possibly imagine doing it. Well... actually evolution doesn't tend to do that well against an engineered process. It has higher bandwidth than human engineers. For example, engineers can move humans around faster and with better energy efficiency [0] than evolution managed directly. We've also figured out more effective ways to organise society (laws & principles) than nature managed (many of our instincts have effects that lead to measurably stupid outcomes, eg, mob forming instincts are just a disaster). There are quite a lot of examples where it turns out engineering > evolution in a quite strict sense. [0] https://en.wikipedia.org/wiki/Energy_efficiency_in_transport https://en.wikipedia.org/wiki/Energy_efficiency_in_transport
    - jcgrillo 2y ago
      Evolution made the engineers, though. And the politicians. We are just as much a "part of nature" as, say, rodents.
    - skydhash 2y ago
      > There are quite a lot of examples where it turns out engineering > evolution in a quite strict sense. Only if you're considering human agents more important than anything else. When considered the system as a whole, nature seems very efficient.
andrewflnr 2y ago
Building the noise filter is part of the learning. I don't see any reason to believe that it's "structural". Babies aren't known for their stellar information processing.
roenxi 2y ago
Practically it has to be higher than the bitrate of audio-visual data that is presented on a computer. Call that 1MBps for a video stream (way under-calling the amount of data human vision reports I would suggest). That'd put a lower cap of around 50 GB/day of new data, 20 TB/year. Of course, computers can train with more than 2 eyes with one location and perspective. We aren't anywhere near the data cap with current training. Although to be fair I do suspect that most of that data is repetitive, boring and of little use. In my opinion some sort of check for novel data is probably going to be the next big breakthrough in machine learning.
- beeflet 2y ago
  > (way under-calling the amount of data human vision reports I would suggest) It doesn't matter because the amount of meaningful data for learning is accessible at a lower resolution. If someone is born with bad vision such that they effectively see at 1/2 or 1/4th of the resolution of a normal person, It's not like they will grow up to be stupid (as long as they can sit in the front of the class to see the chalkboard). In order for visual quality to impair learning it would need to be pretty bad such that you couldn't make out objects or symbols at a reasonable range.

whiplash451 2y ago

Mentioning the blind as evidence that there is no pre-training based on vision is broken in my opinion. It could very well be that humans do multimodal pretraining (including vision), landing babies with a pre-trained brains that works even without vision. A machine implementation of this is pre-training a self-driving car on vision + lidar then doing real-time inference on vision only.