4 ms·

Got me wondering how this compares with neural efficiency, realizing ofc that there's nothing really apples-to-apples here. Training one of these big models ta

Got me wondering how this compares with neural efficiency, realizing ofc that there's nothing really apples-to-apples here. Training one of these big models takes 100kWh for 1e19 flops, so that's 100k Wh, 360M Ws, or 360MJ or 3.6 1e8J. 1e8Joules/1e19flops = 1e-11J/flop Neurons take 1e-8J/spike.[1] Math check appreciated :) Does seem plausible to think of a single neuron spike (hodgkin-huxley cable model) being modeled with ~1k flops. Though I'm firmly of the opinion that nobody really knows how the brain works.. the neural spike activity could be pure epiphenomenon.. who knows! [1] “Finally, the energy supply to a neuron by ATP is 8.31 × 10−9 J. Meanwhile, integrating the total power with respect to time we will get the consumed electric power, which is 8.75 × 10−9 J. This is more energy than the ATP supplied. The energy efficiency is 105.3%. This is an anomaly…” - 2017 Feb 16 Wang, Xu, Institute for Cognitive Neurodynamics, East China University of Science and Technology https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5337805/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5337805/

whatshisface 5y ago
The neural spike is definitely not an epiphenomenon. The action potential / neurotransmitter release / receptor activation process is understood and can be manipulated with electric probes.
- dekhn 5y ago
  For those who are curious, consciousness is an epiphenomenon (an emergenty property of brains), while neural spikes are just physics. See more: https://en.wikipedia.org/wiki/Neural_correlates_of_consciousness https://en.wikipedia.org/wiki/Neural_correlates_of_conscious...
  - whatshisface 5y ago
    I think it would be better to say something like, "paranoia is an epiphenomenon," when nobody knows what consciousness is.
    - dekhn 5y ago
      No. https://en.wikipedia.org/wiki/Epiphenomenalism https://en.wikipedia.org/wiki/Epiphenomenalism
      - whatshisface 5y ago
        Note that that title ends in "ism," like "Calvinism," or "Evangelicalism."
        posterboy 5y ago
        hence epiphenomenal-ize? I don't thinkso. Maybe it's more like epiphenomalis-m(a), where -is could be a genetive ending. I.e. the idea of epiphenomena.
    - netizen-936824 5y ago
      That isn't quite true https://en.m.wikipedia.org/wiki/Consciousness https://en.m.wikipedia.org/wiki/Consciousness
- pmayrgundter 5y ago
  Sorry, didn't mean it quite like that. It's clear neural spike activity exists as a physical process. I'm suggesting that spiking activity may be an epiphenomena more primary brain functions, i.e. information processing, consciousness, etc.. As far as I know, we're closest to showing information processing in the visual cortex (which is highly linear) and we're still a long way from knowing how it works at a neural level. But maybe someone here can update on this? But much of the cortex is highly recurrent (non-linear) and the idea that it's doing something like sending bits between synapses, encoded in spike timing or something.. well, I think that's highly speculative and has plenty of problems. But even if so, that's just "information processing". I'm personally a fan of electromagnetic theories of consciousness[], where the synaptic activity could be an epiphenomenon of supporting a stand EM field. []https://en.wikipedia.org/wiki/Electromagnetic_theories_of_consciousness https://en.wikipedia.org/wiki/Electromagnetic_theories_of_co...
  - whatshisface 5y ago
    >But much of the cortex is highly recurrent (non-linear) and the idea that it's doing something like sending bits between synapses, encoded in spike timing or something.. well, I think that's highly speculative and has plenty of problems. I am not sure how much is known about information processing, but it's clear that motor impulses and sensory information are encoded in the spikes. Higher spike frequency = stronger signal. Synapses are how signals are passed from neuron to neuron.
    - pmayrgundter 5y ago
      Ok, that's fair. That's i/o and yes, that's known to be highly linear by the time it gets to the efferent nerves, and makes sense it is before that as well. I think that still leaves the vast majority of the cortex using undefined mechanisms.
      - whatshisface 5y ago
        There's no need to hypothesize a wholly unique central nervous system signalling mechanism when, not only is the signalling mechanism of peripheral nerves understood, central nerves are observed doing the same thing.

whatshisface 5y ago

The neural spike is definitely not an epiphenomenon. The action potential / neurotransmitter release / receptor activation process is understood and can be manipulated with electric probes.

dekhn 5y ago
For those who are curious, consciousness is an epiphenomenon (an emergenty property of brains), while neural spikes are just physics. See more: https://en.wikipedia.org/wiki/Neural_correlates_of_consciousness https://en.wikipedia.org/wiki/Neural_correlates_of_conscious...
- whatshisface 5y ago
  I think it would be better to say something like, "paranoia is an epiphenomenon," when nobody knows what consciousness is.
  - dekhn 5y ago
    No. https://en.wikipedia.org/wiki/Epiphenomenalism https://en.wikipedia.org/wiki/Epiphenomenalism
    - whatshisface 5y ago
      Note that that title ends in "ism," like "Calvinism," or "Evangelicalism."
      - posterboy 5y ago
        hence epiphenomenal-ize? I don't thinkso. Maybe it's more like epiphenomalis-m(a), where -is could be a genetive ending. I.e. the idea of epiphenomena.
  - netizen-936824 5y ago
    That isn't quite true https://en.m.wikipedia.org/wiki/Consciousness https://en.m.wikipedia.org/wiki/Consciousness
pmayrgundter 5y ago
Sorry, didn't mean it quite like that. It's clear neural spike activity exists as a physical process. I'm suggesting that spiking activity may be an epiphenomena more primary brain functions, i.e. information processing, consciousness, etc.. As far as I know, we're closest to showing information processing in the visual cortex (which is highly linear) and we're still a long way from knowing how it works at a neural level. But maybe someone here can update on this? But much of the cortex is highly recurrent (non-linear) and the idea that it's doing something like sending bits between synapses, encoded in spike timing or something.. well, I think that's highly speculative and has plenty of problems. But even if so, that's just "information processing". I'm personally a fan of electromagnetic theories of consciousness[], where the synaptic activity could be an epiphenomenon of supporting a stand EM field. []https://en.wikipedia.org/wiki/Electromagnetic_theories_of_consciousness https://en.wikipedia.org/wiki/Electromagnetic_theories_of_co...
- whatshisface 5y ago
  >But much of the cortex is highly recurrent (non-linear) and the idea that it's doing something like sending bits between synapses, encoded in spike timing or something.. well, I think that's highly speculative and has plenty of problems. I am not sure how much is known about information processing, but it's clear that motor impulses and sensory information are encoded in the spikes. Higher spike frequency = stronger signal. Synapses are how signals are passed from neuron to neuron.
  - pmayrgundter 5y ago
    Ok, that's fair. That's i/o and yes, that's known to be highly linear by the time it gets to the efferent nerves, and makes sense it is before that as well. I think that still leaves the vast majority of the cortex using undefined mechanisms.
    - whatshisface 5y ago
      There's no need to hypothesize a wholly unique central nervous system signalling mechanism when, not only is the signalling mechanism of peripheral nerves understood, central nerves are observed doing the same thing.