4 ms·

It's weird to me how the article in the very first paragraph mentions brute force, but as data problem. As if we hadn't seen a staggering raise in ops and memor

by lolc 2y ago

It's weird to me how the article in the very first paragraph mentions brute force, but as data problem. As if we hadn't seen a staggering raise in ops and memory bandwidth. Machines have been able to hold the textual history of humans in short-term memory for a while. But the ops they could perform on that have been limited. Not much point telling a 2005 person to "add data". What were they going to do? Wait 20 years to finish a round we do in a week now? It's very clear to me that the progress we observe in machine intelligence is due to brute processing power. Of course evolution of learning algorithms is important! But the main evolution that drives progress is in the compute. Algorithms can be iterated on that much faster if your generations are that much shorter. Why are all these AI companies falling over each other to buy the best compute per watt humans have ever produced? Because compute is king and our head was optimized by evolution to be very efficient at probabilistic computing. That's where machines are catching up. The mark of intelligence is to not need much data at all.

jcgrillo 2y ago
> The mark of intelligence is to not need much data at all. I think part of the answer might be information filtering. The eye can detect single photons, but by the time that information from 10^16 photons/s entering the eyeball gets to the meat CPU it's been filtered down to something relevant and manageable. And at no part in that pipeline is any component operating at more than like 100Hz. So fine tuning the filters to match the processor--and all the high fidelity sensors--simultaneously sounds like a job for evolutionary search if ever there was one. But this is the wild ass guess of someone who doesn't actually know much about biology or machine learning so take with a big chunk of salt.
- kridsdale3 2y ago
  We're also able to do many kilowatts worth of digital inference equivalent processing, instantaneously, for a couple watts at most in cost.
  - jcgrillo 2y ago
    Right, and we're doing it with highly specialized analog hardware, not the general purpose digital kind. Maybe there's a thread to pull on there as well?
    - Nevermark 2y ago
      Over all, biological neurons, are hardly highly specialized. We do have different types of neurons with some different roles and behavior. But once transistors are arranged in circuits, they become highly specialized too. Worse, the vast (vast, vast) majority of all neuron types' structure and activity is not performing computation. Loads of mechanisms and activity aimed at simply handling the complex per neuron energy management systems, continually rebuilding internal and boundary structures, replenishing internal components, selectively absorbing resources, breaking down and expelling waste, etc. Neurons have an incredible design given they must grow, differentiate, and work for a lifetime without any intelligent planning or intervention. But transisters are far smaller, leaner, and faster. And their energy requirement continue to decrease, and heat extraction efficiency improves, we will be able to pack them into fully 3D circuits. They will operate many orders of magnitude faster and more efficiently. There was never an inate benefit to being organic, except that they were achievable with evolution. Digital circuits had to wait for a different kind of ecosystem to originate. But once they appeared, it only took them 100 years to rival their multi-billion year old analog counterparts. Let's not even bring up quantum circuits, which are getting more practical every year, and will outclass both analog and digital circuits for many machine learning algorithms.
      - jcgrillo 2y ago
        It's not so much about the neurons, but the systems they connect to. And how they're organized. And how many of them (relatively) are in each component of the brain. And the feedback between motor control and sensory apparatus. I doubt you'll ever get to "intelligence" by building a really fast "brain in a box", even if it has super great transistors.

by lolc 2y ago

jcgrillo 2y ago

> The mark of intelligence is to not need much data at all. I think part of the answer might be information filtering. The eye can detect single photons, but by the time that information from 10^16 photons/s entering the eyeball gets to the meat CPU it's been filtered down to something relevant and manageable. And at no part in that pipeline is any component operating at more than like 100Hz. So fine tuning the filters to match the processor--and all the high fidelity sensors--simultaneously sounds like a job for evolutionary search if ever there was one. But this is the wild ass guess of someone who doesn't actually know much about biology or machine learning so take with a big chunk of salt.

kridsdale3 2y ago
We're also able to do many kilowatts worth of digital inference equivalent processing, instantaneously, for a couple watts at most in cost.
- jcgrillo 2y ago
  Right, and we're doing it with highly specialized analog hardware, not the general purpose digital kind. Maybe there's a thread to pull on there as well?
  - Nevermark 2y ago
    Over all, biological neurons, are hardly highly specialized. We do have different types of neurons with some different roles and behavior. But once transistors are arranged in circuits, they become highly specialized too. Worse, the vast (vast, vast) majority of all neuron types' structure and activity is not performing computation. Loads of mechanisms and activity aimed at simply handling the complex per neuron energy management systems, continually rebuilding internal and boundary structures, replenishing internal components, selectively absorbing resources, breaking down and expelling waste, etc. Neurons have an incredible design given they must grow, differentiate, and work for a lifetime without any intelligent planning or intervention. But transisters are far smaller, leaner, and faster. And their energy requirement continue to decrease, and heat extraction efficiency improves, we will be able to pack them into fully 3D circuits. They will operate many orders of magnitude faster and more efficiently. There was never an inate benefit to being organic, except that they were achievable with evolution. Digital circuits had to wait for a different kind of ecosystem to originate. But once they appeared, it only took them 100 years to rival their multi-billion year old analog counterparts. Let's not even bring up quantum circuits, which are getting more practical every year, and will outclass both analog and digital circuits for many machine learning algorithms.
    - jcgrillo 2y ago
      It's not so much about the neurons, but the systems they connect to. And how they're organized. And how many of them (relatively) are in each component of the brain. And the feedback between motor control and sensory apparatus. I doubt you'll ever get to "intelligence" by building a really fast "brain in a box", even if it has super great transistors.