I don't think Relus are the proplem. Relus are good, but artificial neural networks can work with sigmoid activation fonctions instead. The neurones in an ANN w

I don't think Relus are the proplem. Relus are good, but artificial neural networks can work with sigmoid activation fonctions instead. The neurones in an ANN with sigmoids are somewhat close to what is described in this article. The main problem is that the neurons in this article are not trainable (you can't update the weights dynamically) and not differentiable (you can't use the usual gradient-based training procedures to update the weights). They could be used for inference only though.