3 ms·

This paper is concerning. While divorced from the standard ML literature there is a lot of work on biologically plausible spiking, timing dependant artificial n

This paper is concerning. While divorced from the standard ML literature there is a lot of work on biologically plausible spiking, timing dependant artificial neutral networks. The nomenclature here doesn't seem to acknowledge that body of work. Instead it appears as a step toward that bulk of research coming from the ML/LLM field without a clear appreciation of the ground well traveled there.* In addition some of the terminology is likely to cause confusion. By calling a synaptic integration step "thinking" the authors are going to confuse a lot of people. Instead of the process of forming an idea, evaluating that idea, potentially modifying it and repeating (what a layman would call thinking) they are trying to ascribe "thinking" to single unit processes! That's a pretty radical departure from both ML and ANN literature. Pattern recognition/signal discrimination is well known at the level of synaptic integration and firing, but "thinking?" No, that wording is not helpful. *I have not reviewed all the citations and am reacting to the plain language of the text as someone familiar with both lines of research.

cepera 1y ago
>there is a lot of work on biologically plausible spiking I ask you kindly to share the list (or even better brief review) of most insightful books/papers in your opinion with neuroscience inspired algorithms concepts/implementation details.
- erewhile 1y ago
  Not the original poster, but: - Theoretical Neuroscience Computational and Mathematical Modeling of Neural Systems - Peter Dayan, L. F. Abbott (2001) is quite good, more mathematical than computational. - Neuronal dynamics, available here: https://neuronaldynamics.epfl.ch/ https://neuronaldynamics.epfl.ch/ is also quite good, and free to read. Has python exercises as well. If I recall correctly, it mostly goes into simulations of singular neurons, and not so much entire networks and what we can do with them, but it does a good job at bridging the chemistry / biology / math to computation. If we're talking about papers, one I mentioned in my other comment: - Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations, https://doi.org/10.1162/089976602760407955 https://doi.org/10.1162/089976602760407955 - Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons, by Nicolas Brunel (Don't have a DOI on hand for this one) - Spiking Neural Networks and Their Applications: A Review, https://doi.org/10.3390/brainsci12070863 https://doi.org/10.3390/brainsci12070863 , is a very nice review of methods and does some nice explaining on concepts. If you're looking for keywords on the topic: - Leaky Integrate and Fire (LIF) neurons - Spiking neural networks - Liquid State Machines (LSM) - Synaptic plasticity (Models of synaptic plasticity) - Spike-based synaptic plasticity
- rkp8000 1y ago
  A (non-exhaustive) list of some notable papers: Maass 2002, Real-time computing without stable states: https://pubmed.ncbi.nlm.nih.gov/12433288/ https://pubmed.ncbi.nlm.nih.gov/12433288/ Sussillo & Abbott 2009, Generating Coherent Patterns of Activity from Chaotic Neural Networks https://pmc.ncbi.nlm.nih.gov/articles/PMC2756108/ https://pmc.ncbi.nlm.nih.gov/articles/PMC2756108/ Abbott et al 2016, Building functional networks of spiking model neurons https://pubmed.ncbi.nlm.nih.gov/26906501/ https://pubmed.ncbi.nlm.nih.gov/26906501/ Zenke & Ganguli 2018, SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks https://ganguli-gang.stanford.edu/pdf/17.superspike.pdf https://ganguli-gang.stanford.edu/pdf/17.superspike.pdf Bellec et al 2020, A solution to the learning dilemma for recurrent networks of spiking neurons https://www.nature.com/articles/s41467-020-17236-y https://www.nature.com/articles/s41467-020-17236-y Payeur et al 2021, Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits https://www.nature.com/articles/s41593-021-00857-x https://www.nature.com/articles/s41593-021-00857-x Cimesa et al 2023, Geometry of population activity in spiking networks with low-rank structure https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011315 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... Ororbia 2024, Contrastive signal–dependent plasticity: Self-supervised learning in spiking neural circuits https://www.science.org/doi/10.1126/sciadv.adn6076 https://www.science.org/doi/10.1126/sciadv.adn6076 Kudithipudi et al 2025, Neuromorphic computing at scale (review) https://www.nature.com/articles/s41586-024-08253-8 https://www.nature.com/articles/s41586-024-08253-8
tiahura 1y ago
The authors don't label a single synaptic integration as "thinking." They use the term for the network-wide internal loop ("internal ticks") that unrolls after every external input, and explicitly say it is merely "analogous to thought."
vonneumannstan 1y ago
Was this written by Jürgen Schmidhuber?
mountainriver 1y ago
Agree, they are presenting this like its a new idea without hardly any reference to the decades of work on spiking neural nets and similar.
TeMPOraL 1y ago
I'm sort of not surprised; my impression is that, for the past decade or two, ML researchers who did acknowledge related work in neuroscience were broadly accused of hubris for daring to compare their work to biological brains.
program_whiz 1y ago
Sorry I should have responded to this comment, but I wrote a separate response in the parent thread. I didn't feel the pdf / paper was really trying to mimick spiking biological networks in all but the loosest sense (there is a sequence of activations and layers of "neurons"). I think the major contribution is just using the dot product on output transpose output, the rest is just diffusion / attention on inputs. Its conceptually a combination of "input attention" and "output attention" using a kind of stepped recursive model.

cepera 1y ago

>there is a lot of work on biologically plausible spiking I ask you kindly to share the list (or even better brief review) of most insightful books/papers in your opinion with neuroscience inspired algorithms concepts/implementation details.

erewhile 1y ago
Not the original poster, but: - Theoretical Neuroscience Computational and Mathematical Modeling of Neural Systems - Peter Dayan, L. F. Abbott (2001) is quite good, more mathematical than computational. - Neuronal dynamics, available here: https://neuronaldynamics.epfl.ch/ https://neuronaldynamics.epfl.ch/ is also quite good, and free to read. Has python exercises as well. If I recall correctly, it mostly goes into simulations of singular neurons, and not so much entire networks and what we can do with them, but it does a good job at bridging the chemistry / biology / math to computation. If we're talking about papers, one I mentioned in my other comment: - Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations, https://doi.org/10.1162/089976602760407955 https://doi.org/10.1162/089976602760407955 - Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons, by Nicolas Brunel (Don't have a DOI on hand for this one) - Spiking Neural Networks and Their Applications: A Review, https://doi.org/10.3390/brainsci12070863 https://doi.org/10.3390/brainsci12070863 , is a very nice review of methods and does some nice explaining on concepts. If you're looking for keywords on the topic: - Leaky Integrate and Fire (LIF) neurons - Spiking neural networks - Liquid State Machines (LSM) - Synaptic plasticity (Models of synaptic plasticity) - Spike-based synaptic plasticity
rkp8000 1y ago
A (non-exhaustive) list of some notable papers: Maass 2002, Real-time computing without stable states: https://pubmed.ncbi.nlm.nih.gov/12433288/ https://pubmed.ncbi.nlm.nih.gov/12433288/ Sussillo & Abbott 2009, Generating Coherent Patterns of Activity from Chaotic Neural Networks https://pmc.ncbi.nlm.nih.gov/articles/PMC2756108/ https://pmc.ncbi.nlm.nih.gov/articles/PMC2756108/ Abbott et al 2016, Building functional networks of spiking model neurons https://pubmed.ncbi.nlm.nih.gov/26906501/ https://pubmed.ncbi.nlm.nih.gov/26906501/ Zenke & Ganguli 2018, SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks https://ganguli-gang.stanford.edu/pdf/17.superspike.pdf https://ganguli-gang.stanford.edu/pdf/17.superspike.pdf Bellec et al 2020, A solution to the learning dilemma for recurrent networks of spiking neurons https://www.nature.com/articles/s41467-020-17236-y https://www.nature.com/articles/s41467-020-17236-y Payeur et al 2021, Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits https://www.nature.com/articles/s41593-021-00857-x https://www.nature.com/articles/s41593-021-00857-x Cimesa et al 2023, Geometry of population activity in spiking networks with low-rank structure https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011315 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... Ororbia 2024, Contrastive signal–dependent plasticity: Self-supervised learning in spiking neural circuits https://www.science.org/doi/10.1126/sciadv.adn6076 https://www.science.org/doi/10.1126/sciadv.adn6076 Kudithipudi et al 2025, Neuromorphic computing at scale (review) https://www.nature.com/articles/s41586-024-08253-8 https://www.nature.com/articles/s41586-024-08253-8

tiahura 1y ago

The authors don't label a single synaptic integration as "thinking." They use the term for the network-wide internal loop ("internal ticks") that unrolls after every external input, and explicitly say it is merely "analogous to thought."

vonneumannstan 1y ago

Was this written by Jürgen Schmidhuber?

mountainriver 1y ago

Agree, they are presenting this like its a new idea without hardly any reference to the decades of work on spiking neural nets and similar.

TeMPOraL 1y ago

I'm sort of not surprised; my impression is that, for the past decade or two, ML researchers who did acknowledge related work in neuroscience were broadly accused of hubris for daring to compare their work to biological brains.

program_whiz 1y ago

Sorry I should have responded to this comment, but I wrote a separate response in the parent thread. I didn't feel the pdf / paper was really trying to mimick spiking biological networks in all but the loosest sense (there is a sequence of activations and layers of "neurons"). I think the major contribution is just using the dot product on output transpose output, the rest is just diffusion / attention on inputs. Its conceptually a combination of "input attention" and "output attention" using a kind of stepped recursive model.