When I read "Dead Neuron" I definitely associated it with ReLUs. It's claimed that this tendency to create dead neurons has a regularizing effect, but frankly

When I read "Dead Neuron" I definitely associated it with ReLUs. It's claimed that this tendency to create dead neurons has a regularizing effect, but frankly I don't buy that. Anecdotally, I've always had better results with Leaky ReLUs or Mish.