4 ms·
Actually ReLU(x) = max(x,0) can be thought after a change of parameters as the firing rate response of a leaky integrate and fire neuron at least for a certain
by orbifold 9y ago
Actually ReLU(x) = max(x,0) can be thought after a change of parameters as the firing rate response of a leaky integrate and fire neuron at least for a certain range of parameters. So it is not completely far fetched, but of course nature does not use rate coding exclusively.
- eru 9y agoYes. Though I think they stumbled on ReLU just from technical considerations and not taking nature into account? That happens fairly often in engineering: looking around enough one can often find natural precedents for our artifices. But directly trying to copy nature did not lead to the plane or helicopter.
- zardo 9y agoWikipedia list this as the first paper involving ReLU in an ANN: [1] R Hahnloser, R. Sarpeshkar, M A Mahowald, R. J. Douglas, H.S. Seung (2000). Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.