3 ms·

I don't think they understimate the non-linearity of real neurons. It's simply a trade-off. Accurate models were unpractical (too computationally expensive) the

by slx26 6y ago

I don't think they understimate the non-linearity of real neurons. It's simply a trade-off. Accurate models were unpractical (too computationally expensive) the last time I checked. There's quite a bit of research on neuron modelling, both for accurate responses and approximations that are fast to compute. The wikipedia article [0] is actually a really good read. What's true is that with the current upscaling of neural networks, it might be worth it to look back at more sophisticate approximations and see if they are implementable now. But I guess the problem would not be just training, but the fact that the trained networks would also be more expensive to run. I don't really know much about the topic, so if anyone has studied this in more detail and wants to share something... [0] https://en.wikipedia.org/wiki/Biological_neuron_model https://en.wikipedia.org/wiki/Biological_neuron_model

a-priori 6y ago
It's tricky to be both reasonably biologically accurate, and yet computationally efficient. Some models do exist to fit this niche. One I'm familiar with is Izhikevich (2003), a model with four morphology parameters and two state variables and corresponding ordinary differential equations. It can simulate the spiking behaviour of several different kinds of neurons. https://www.izhikevich.org/publications/spikes.htm https://www.izhikevich.org/publications/spikes.htm I've written a SIMD-based simulator that can integrate 500k of those neurons in real-time on a standard laptop. That's a bit of a limit case -- it doesn't yet model action potentials or learning, which would slow it down quite a bit. The details here matter a lot: a GPU version I experimented with could do about 1M, whereas an earlier version of this work that was a straight-up CPU-based solution (_not_ SIMD) but _with_ action potentials and learning could only model 1k.
- slx26 6y ago
  That's very interesting, thanks for sharing!

by slx26 6y ago

a-priori 6y ago

It's tricky to be both reasonably biologically accurate, and yet computationally efficient. Some models do exist to fit this niche. One I'm familiar with is Izhikevich (2003), a model with four morphology parameters and two state variables and corresponding ordinary differential equations. It can simulate the spiking behaviour of several different kinds of neurons. https://www.izhikevich.org/publications/spikes.htm https://www.izhikevich.org/publications/spikes.htm I've written a SIMD-based simulator that can integrate 500k of those neurons in real-time on a standard laptop. That's a bit of a limit case -- it doesn't yet model action potentials or learning, which would slow it down quite a bit. The details here matter a lot: a GPU version I experimented with could do about 1M, whereas an earlier version of this work that was a straight-up CPU-based solution (_not_ SIMD) but _with_ action potentials and learning could only model 1k.

slx26 6y ago
That's very interesting, thanks for sharing!