3 ms·

You're right on the face of it, the added context that's missing is Numenta has been pushing bio-inspired AI for like 17 years and nothing practical has come of

You're right on the face of it, the added context that's missing is Numenta has been pushing bio-inspired AI for like 17 years and nothing practical has come of it. If using some energy to achieve a practical goal is wasteful, using less energy to accomplish nothing is more so. I got into AI after a grad school career split between mathematical signal processing and computational neuroscience. I knew folks back in the early 2010's looking at joining Numenta. The ideas are absolutely good to explore, it's the execution that's lacking. Maybe their big breakthrough is just around the corner, but how long do you wait for product 1 alpha build 1 before calling out vaporware?

photochemsyn 4y ago
That's valid. As I was reading through it I was wondering if their claims about how brains actually work internally were all that well supported, things like: A biological neuron has two kinds of dendrites: distal and proximal. Only proximal dendrites are modeled in the artificial neurons we see today. Nerve cells in the brain also have what, ~20 different chemical neurotransmitters modulating the state of the neuron in question, and this looks a lot more analog than digital... any kind of one-to-one correspondence between biological brains and digital learning networks never seemed all that realistic to me.
- heyitsguay 4y ago
  I don't think their scientific claims are all that inaccurate, maybe some stuff is simplified but that's inevitable. To my understanding, the issue is more converting that advanced scientific knowledge into something tangible. Can I build a Google photos competitor using spiking neural nets for content recognition? Can i train GPT-4 with 50% of the power bill of GPT-3? Can i dare hope for better accuracy or robustness for some production task? Or better compressed nets for edge computing? I'm not seeing anything like that. We've known a long time that ANNs as implemented today are very crude compared to biological neurons; they really just share a name. What's supposed to differentiate a company from a lab is taking that insight and doing something concrete with it.
- codebolt 4y ago
  A neuron in the brain is more like it's own microprocessor than just a memory cell anyway, which is why the current style of ANNs ('bio-inspired' or not) will never come close to the computational complexity involved in anything like a mammalian brain.
- l33tman 4y ago
  Yes he is not keeping current. Modern AI is differentiable functions, taking about "neurons" and their mapping to brain hardware at this point is old school at best.
evrydayhustling 4y ago
Additionally, the ideas here are well represented in mainstream deep learning research by the supposed energy offenders. For example, here are links representing practical applications using sparse inference, continuous multitask learning, and specialized hardware -- just by Google. The article is marketing to folks who care about the environment, but have limited background on modern ML (and perhaps also energy economics). [1] https://ai.googleblog.com/2021/03/accelerating-neural-networks-on-mobile.html https://ai.googleblog.com/2021/03/accelerating-neural-networ... [2] https://ai.googleblog.com/2022/03/robust-graph-neural-networks.html https://ai.googleblog.com/2022/03/robust-graph-neural-networ... [3] https://ai.googleblog.com/2021/04/multi-task-robotic-reinforcement.html https://ai.googleblog.com/2021/04/multi-task-robotic-reinfor... [4] https://cloud.google.com/tpu/docs/tpus https://cloud.google.com/tpu/docs/tpus

photochemsyn 4y ago

That's valid. As I was reading through it I was wondering if their claims about how brains actually work internally were all that well supported, things like: A biological neuron has two kinds of dendrites: distal and proximal. Only proximal dendrites are modeled in the artificial neurons we see today. Nerve cells in the brain also have what, ~20 different chemical neurotransmitters modulating the state of the neuron in question, and this looks a lot more analog than digital... any kind of one-to-one correspondence between biological brains and digital learning networks never seemed all that realistic to me.

heyitsguay 4y ago
I don't think their scientific claims are all that inaccurate, maybe some stuff is simplified but that's inevitable. To my understanding, the issue is more converting that advanced scientific knowledge into something tangible. Can I build a Google photos competitor using spiking neural nets for content recognition? Can i train GPT-4 with 50% of the power bill of GPT-3? Can i dare hope for better accuracy or robustness for some production task? Or better compressed nets for edge computing? I'm not seeing anything like that. We've known a long time that ANNs as implemented today are very crude compared to biological neurons; they really just share a name. What's supposed to differentiate a company from a lab is taking that insight and doing something concrete with it.
codebolt 4y ago
A neuron in the brain is more like it's own microprocessor than just a memory cell anyway, which is why the current style of ANNs ('bio-inspired' or not) will never come close to the computational complexity involved in anything like a mammalian brain.
l33tman 4y ago
Yes he is not keeping current. Modern AI is differentiable functions, taking about "neurons" and their mapping to brain hardware at this point is old school at best.

evrydayhustling 4y ago

Additionally, the ideas here are well represented in mainstream deep learning research by the supposed energy offenders. For example, here are links representing practical applications using sparse inference, continuous multitask learning, and specialized hardware -- just by Google. The article is marketing to folks who care about the environment, but have limited background on modern ML (and perhaps also energy economics). [1] https://ai.googleblog.com/2021/03/accelerating-neural-networks-on-mobile.html https://ai.googleblog.com/2021/03/accelerating-neural-networ... [2] https://ai.googleblog.com/2022/03/robust-graph-neural-networks.html https://ai.googleblog.com/2022/03/robust-graph-neural-networ... [3] https://ai.googleblog.com/2021/04/multi-task-robotic-reinforcement.html https://ai.googleblog.com/2021/04/multi-task-robotic-reinfor... [4] https://cloud.google.com/tpu/docs/tpus https://cloud.google.com/tpu/docs/tpus